Sep 08 11:28:21.484714 ip-10-0-133-187 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 08 11:28:21.484724 ip-10-0-133-187 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 08 11:28:21.484731 ip-10-0-133-187 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 08 11:28:21.484888 ip-10-0-133-187 systemd[1]: Failed to start Kubernetes Kubelet. Sep 08 11:28:31.598736 ip-10-0-133-187 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 08 11:28:31.598752 ip-10-0-133-187 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7677380cf03b41f2b825148e8430d5de -- Sep 08 11:30:09.909000 ip-10-0-133-187 systemd[1]: Starting Kubernetes Kubelet... Sep 08 11:30:10.285523 ip-10-0-133-187 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 11:30:10.285523 ip-10-0-133-187 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 08 11:30:10.285523 ip-10-0-133-187 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 11:30:10.285523 ip-10-0-133-187 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 08 11:30:10.285523 ip-10-0-133-187 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 11:30:10.287577 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.287498 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 08 11:30:10.294602 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294580 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 11:30:10.294602 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294597 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 11:30:10.294602 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294601 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 11:30:10.294602 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294605 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 11:30:10.294602 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294608 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 11:30:10.294602 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294611 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294614 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294617 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294620 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294622 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294625 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294628 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294631 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294633 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294636 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294638 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294641 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294643 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294646 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294649 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294652 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294654 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294657 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294659 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294669 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 11:30:10.294833 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294671 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294676 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294679 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294682 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294684 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294687 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294690 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294693 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294695 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294698 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294700 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294703 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294706 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294709 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294713 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294715 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294730 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294733 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294736 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 11:30:10.295319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294739 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294742 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294745 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294747 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294750 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294752 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294755 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294758 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294761 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294763 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294765 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294768 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294770 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294773 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294775 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294778 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294781 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294784 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294786 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294788 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 11:30:10.295805 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294792 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294794 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294797 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294799 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294802 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294804 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294808 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294811 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294814 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294817 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294819 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294822 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294824 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294827 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294829 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294832 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294835 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294837 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294841 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294844 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 11:30:10.296281 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294847 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294849 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294852 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294856 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294860 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294862 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294865 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294868 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294872 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294875 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294878 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294881 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294884 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294887 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.294890 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295589 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295596 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295599 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295603 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 11:30:10.296818 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295606 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295608 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295611 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295614 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295616 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295619 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295622 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295624 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295627 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295630 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295633 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295636 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295638 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295642 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295645 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295648 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295650 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295653 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295655 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295658 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 11:30:10.297280 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295660 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295663 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295665 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295668 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295670 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295673 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295675 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295678 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295681 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295683 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295686 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295689 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295692 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295695 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295697 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295700 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295703 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295706 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295708 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295711 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 11:30:10.297806 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295713 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295716 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295740 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295743 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295746 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295748 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295751 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295754 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295757 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295760 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295762 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295765 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295767 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295770 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295773 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295775 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295778 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295780 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295783 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295785 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 11:30:10.298295 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295788 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295790 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295794 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295799 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295803 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295806 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295808 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295813 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295816 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295819 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295821 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295824 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295827 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295829 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295832 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295834 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295837 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295839 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295842 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 11:30:10.298792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295845 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295849 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295853 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295859 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295862 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295865 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295868 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295871 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295874 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295877 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295880 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295882 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295885 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295887 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295890 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.295892 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296284 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296291 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296294 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 11:30:10.299283 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296297 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296300 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296303 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296306 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296308 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296311 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296314 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296317 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296319 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296321 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296324 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296327 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296329 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296332 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296335 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296337 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296340 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296342 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296345 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296348 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 11:30:10.299764 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296351 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296353 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296356 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296359 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296361 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296364 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296366 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296369 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296372 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296375 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296378 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296381 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296383 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296386 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296388 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296391 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296393 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296395 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296398 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296401 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 11:30:10.300249 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296403 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296406 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296408 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296410 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296413 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296415 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296418 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296421 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296423 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296426 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296428 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296431 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296433 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296436 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296438 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296441 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296446 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296449 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296452 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296455 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 11:30:10.300794 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296458 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296461 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296463 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296466 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296469 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296472 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296474 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296477 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296479 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296482 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296484 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296487 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296489 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296492 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296494 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296497 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296499 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296503 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296507 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 11:30:10.301286 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296510 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296513 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296516 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296518 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296521 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296524 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296526 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296529 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296531 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296534 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296536 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296539 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296541 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296544 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296547 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296549 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.296552 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296619 2564 flags.go:64] FLAG: --address="0.0.0.0" Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296649 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296656 2564 flags.go:64] FLAG: --anonymous-auth="true" Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296661 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 08 11:30:10.301849 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296665 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296670 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296674 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296679 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296682 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296685 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296688 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296692 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296695 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296698 2564 flags.go:64] FLAG: --cgroup-root="" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296701 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296705 2564 flags.go:64] FLAG: --client-ca-file="" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296708 2564 flags.go:64] FLAG: --cloud-provider="external" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296711 2564 flags.go:64] FLAG: --cluster-dns="[]" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296715 2564 flags.go:64] FLAG: --cluster-domain="" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296734 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296738 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296741 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296745 2564 flags.go:64] FLAG: --container-log-max-files="5" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296748 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296751 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296754 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296758 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296761 2564 flags.go:64] FLAG: --contention-profiling="false" Sep 08 11:30:10.302355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296764 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296767 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296770 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296773 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296777 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296780 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296783 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296785 2564 flags.go:64] FLAG: --enable-load-reader="false" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296788 2564 flags.go:64] FLAG: --enable-server="true" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296791 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296796 2564 flags.go:64] FLAG: --event-burst="100" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296799 2564 flags.go:64] FLAG: --event-qps="50" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296802 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296805 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296808 2564 flags.go:64] FLAG: --eviction-hard="" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296811 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296814 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296817 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296820 2564 flags.go:64] FLAG: --eviction-soft="" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296823 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296826 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296830 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296833 2564 flags.go:64] FLAG: --experimental-mounter-path="" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296837 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296840 2564 flags.go:64] FLAG: --fail-swap-on="true" Sep 08 11:30:10.302976 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296843 2564 flags.go:64] FLAG: --feature-gates="" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296847 2564 flags.go:64] FLAG: --file-check-frequency="20s" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296850 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296853 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296856 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296859 2564 flags.go:64] FLAG: --healthz-port="10248" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296863 2564 flags.go:64] FLAG: --help="false" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296866 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296869 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296873 2564 flags.go:64] FLAG: --http-check-frequency="20s" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296875 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296879 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296882 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296885 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296888 2564 flags.go:64] FLAG: --image-service-endpoint="" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296891 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296894 2564 flags.go:64] FLAG: --kube-api-burst="100" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296897 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296900 2564 flags.go:64] FLAG: --kube-api-qps="50" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296903 2564 flags.go:64] FLAG: --kube-reserved="" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296906 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296909 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296912 2564 flags.go:64] FLAG: --kubelet-cgroups="" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296915 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 08 11:30:10.303608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296918 2564 flags.go:64] FLAG: --lock-file="" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296921 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296924 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296927 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296932 2564 flags.go:64] FLAG: --log-json-split-stream="false" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296936 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296940 2564 flags.go:64] FLAG: --log-text-split-stream="false" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296942 2564 flags.go:64] FLAG: --logging-format="text" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296945 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296949 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296952 2564 flags.go:64] FLAG: --manifest-url="" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296955 2564 flags.go:64] FLAG: --manifest-url-header="" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296959 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296962 2564 flags.go:64] FLAG: --max-open-files="1000000" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296968 2564 flags.go:64] FLAG: --max-pods="110" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296971 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296974 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296977 2564 flags.go:64] FLAG: --memory-manager-policy="None" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296980 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296983 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296986 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296989 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.296996 2564 flags.go:64] FLAG: --node-status-max-images="50" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297000 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297003 2564 flags.go:64] FLAG: --oom-score-adj="-999" Sep 08 11:30:10.304232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297006 2564 flags.go:64] FLAG: --pod-cidr="" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297009 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297014 2564 flags.go:64] FLAG: --pod-manifest-path="" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297017 2564 flags.go:64] FLAG: --pod-max-pids="-1" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297020 2564 flags.go:64] FLAG: --pods-per-core="0" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297023 2564 flags.go:64] FLAG: --port="10250" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297026 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297030 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c138c386b9a87c1b" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297033 2564 flags.go:64] FLAG: --qos-reserved="" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297036 2564 flags.go:64] FLAG: --read-only-port="10255" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297039 2564 flags.go:64] FLAG: --register-node="true" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297041 2564 flags.go:64] FLAG: --register-with-taints="" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297045 2564 flags.go:64] FLAG: --registry-burst="10" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297048 2564 flags.go:64] FLAG: --registry-qps="5" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297052 2564 flags.go:64] FLAG: --reserved-cpus="" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297056 2564 flags.go:64] FLAG: --reserved-memory="" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297059 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297063 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297066 2564 flags.go:64] FLAG: --rotate-certificates="false" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297069 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297071 2564 flags.go:64] FLAG: --runonce="false" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297076 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297079 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297082 2564 flags.go:64] FLAG: --seccomp-default="false" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297085 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 08 11:30:10.304859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297088 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297091 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297094 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297097 2564 flags.go:64] FLAG: --storage-driver-password="root" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297100 2564 flags.go:64] FLAG: --storage-driver-secure="false" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297103 2564 flags.go:64] FLAG: --storage-driver-table="stats" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297106 2564 flags.go:64] FLAG: --storage-driver-user="root" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297109 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297112 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297115 2564 flags.go:64] FLAG: --system-cgroups="" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297118 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297123 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297126 2564 flags.go:64] FLAG: --tls-cert-file="" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297129 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297133 2564 flags.go:64] FLAG: --tls-min-version="" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297136 2564 flags.go:64] FLAG: --tls-private-key-file="" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297138 2564 flags.go:64] FLAG: --topology-manager-policy="none" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297141 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297144 2564 flags.go:64] FLAG: --topology-manager-scope="container" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297147 2564 flags.go:64] FLAG: --v="2" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297151 2564 flags.go:64] FLAG: --version="false" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297156 2564 flags.go:64] FLAG: --vmodule="" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297160 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297164 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297288 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 11:30:10.305516 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297293 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297296 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297299 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297304 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297307 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297309 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297312 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297315 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297318 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297321 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297323 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297326 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297329 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297333 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297336 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297339 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297342 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297345 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297347 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 11:30:10.306138 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297350 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297352 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297355 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297357 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297360 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297362 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297365 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297368 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297370 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297374 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297377 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297381 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297383 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297386 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297388 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297391 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297395 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297397 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297400 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297409 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 11:30:10.306610 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297413 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297416 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297419 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297421 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297424 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297426 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297429 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297431 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297434 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297436 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297439 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297441 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297444 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297446 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297449 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297451 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297454 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297457 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297460 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297464 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 11:30:10.307107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297467 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297470 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297473 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297477 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297481 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297485 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297489 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297491 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297496 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297500 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297504 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297507 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297509 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297512 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297515 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297518 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297521 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297523 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297526 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 11:30:10.307582 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297529 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297532 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297534 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297537 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297540 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297542 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297544 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297547 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297549 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297552 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297554 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297557 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297559 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297562 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297570 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297573 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297575 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297578 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297581 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 11:30:10.308125 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.297584 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 11:30:10.308585 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.297588 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 11:30:10.308585 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.305045 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 08 11:30:10.308585 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.305061 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 08 11:30:10.308585 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.305987 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 08 11:30:10.308585 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.305998 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 08 11:30:10.308585 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306056 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 11:30:10.308585 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306062 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 11:30:10.308585 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306066 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 11:30:10.308585 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306069 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 11:30:10.308585 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306072 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 11:30:10.308585 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306075 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 11:30:10.308585 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306078 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 11:30:10.308585 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306081 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 11:30:10.308585 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306086 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306090 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306093 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306096 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306099 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306102 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306105 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306107 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306111 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306115 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306118 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306121 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306123 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306126 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306128 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306131 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306134 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306136 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306139 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306142 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 11:30:10.308992 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306145 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306149 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306152 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306155 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306157 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306161 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306163 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306166 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306169 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306171 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306174 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306176 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306179 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306181 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306184 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306187 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306189 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306192 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306195 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 11:30:10.309484 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306197 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306200 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306202 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306205 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306209 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306212 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306215 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306218 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306221 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306223 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306226 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306229 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306232 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306235 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306238 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306241 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306244 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306246 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306249 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 11:30:10.309958 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306251 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306254 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306256 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306259 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306261 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306264 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306266 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306269 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306271 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306274 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306276 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306279 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306281 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306284 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306286 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306289 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306291 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306294 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306296 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306299 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 11:30:10.310412 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306301 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 11:30:10.310964 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306304 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 11:30:10.310964 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306306 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 11:30:10.310964 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306309 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 11:30:10.310964 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306311 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 11:30:10.310964 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306314 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 11:30:10.310964 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306316 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 11:30:10.310964 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306319 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 11:30:10.310964 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306322 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 11:30:10.310964 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306325 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 11:30:10.310964 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306327 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 11:30:10.310964 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306330 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 08 11:30:10.310964 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306332 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 11:30:10.310964 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.306337 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 11:30:10.310964 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306450 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 11:30:10.310964 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306454 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306457 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306460 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306463 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306466 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306470 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306474 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306477 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306480 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306482 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306485 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306487 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306490 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306492 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306495 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306498 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306500 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306503 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306505 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 11:30:10.311338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306507 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306510 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306513 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306515 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306518 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306520 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306523 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306525 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306528 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306531 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306534 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306536 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306539 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306541 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306544 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306546 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306549 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306551 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306555 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306558 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 11:30:10.311914 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306561 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306563 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306566 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306568 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306571 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306573 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306576 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306579 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306581 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306584 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306586 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306589 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306592 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306594 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306596 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306599 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306601 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306605 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306608 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 11:30:10.312408 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306611 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306614 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306618 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306621 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306623 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306626 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306629 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306632 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306634 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306652 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306656 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306659 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306662 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306664 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306667 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306669 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306672 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306674 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306677 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306679 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 11:30:10.312982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306682 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306685 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306687 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306690 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306692 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306695 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306697 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306700 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306702 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306705 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306707 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306710 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306713 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306715 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306718 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306736 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306739 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306742 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306744 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 11:30:10.313488 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:10.306746 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 08 11:30:10.314094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.306751 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 11:30:10.314094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.306857 2564 server.go:962] "Client rotation is on, will bootstrap in background" Sep 08 11:30:10.314094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.310407 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 08 11:30:10.314094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.311466 2564 server.go:1019] "Starting client certificate rotation" Sep 08 11:30:10.314094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.311581 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 11:30:10.314094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.311623 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 11:30:10.315308 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.315292 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 11:30:10.319261 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.319244 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 11:30:10.330304 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.330284 2564 log.go:25] "Validated CRI v1 runtime API" Sep 08 11:30:10.333645 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.333629 2564 log.go:25] "Validated CRI v1 image API" Sep 08 11:30:10.334473 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.334458 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 08 11:30:10.337857 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.337841 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c56bb238-b65a-418a-abf8-7818ff59828b:/dev/nvme0n1p3 ffff9e20-c07e-4153-9dc7-b30c0e86f58f:/dev/nvme0n1p4] Sep 08 11:30:10.337903 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.337857 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 08 11:30:10.343066 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.342955 2564 manager.go:216] Machine: {Timestamp:2026-09-08 11:30:10.341511343 +0000 UTC m=+0.334344308 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099953 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25f2821909a84afb1be26c067a016d SystemUUID:ec25f282-1909-a84a-fb1b-e26c067a016d BootID:7677380c-f03b-41f2-b825-148e8430d5de Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4d:3c:5f:fd:f5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4d:3c:5f:fd:f5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:d8:f4:fb:dc:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 08 11:30:10.343066 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.343064 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 08 11:30:10.343171 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.343139 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 08 11:30:10.344079 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.344065 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 11:30:10.346498 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.346474 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 08 11:30:10.346627 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.346501 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-187.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 08 11:30:10.347321 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.347310 2564 topology_manager.go:138] "Creating topology manager with none policy" Sep 08 11:30:10.347359 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.347325 2564 container_manager_linux.go:309] "Creating device plugin manager" Sep 08 11:30:10.347359 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.347332 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 11:30:10.348152 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.348141 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 11:30:10.350162 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.350153 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 08 11:30:10.354700 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.354689 2564 state_mem.go:36] "Initialized new in-memory state store" Sep 08 11:30:10.354824 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.354815 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 08 11:30:10.354868 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.354844 2564 kubelet.go:486] "Attempting to sync node with API server" Sep 08 11:30:10.354868 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.354863 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 08 11:30:10.354936 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.354873 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 08 11:30:10.354936 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.354882 2564 kubelet.go:398] "Adding apiserver pod source" Sep 08 11:30:10.354936 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.354891 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 08 11:30:10.355232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.355220 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 11:30:10.355281 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.355242 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 11:30:10.357788 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.357763 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 08 11:30:10.360764 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.360749 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 08 11:30:10.360834 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.360779 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 08 11:30:10.360950 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.360940 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 08 11:30:10.360982 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.360955 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 08 11:30:10.360982 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.360961 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 08 11:30:10.360982 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.360967 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 08 11:30:10.360982 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.360977 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 08 11:30:10.361090 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.360986 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 08 11:30:10.361595 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.361584 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 08 11:30:10.361595 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.361595 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 08 11:30:10.361658 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.361602 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 08 11:30:10.361658 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.361608 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 08 11:30:10.361658 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.361616 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 08 11:30:10.361658 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.361623 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 08 11:30:10.362527 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.362517 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 08 11:30:10.362527 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.362525 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 08 11:30:10.366277 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.366244 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-187.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 08 11:30:10.366373 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.366358 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 08 11:30:10.366430 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.366389 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 08 11:30:10.366467 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.366435 2564 server.go:1268] "Started kubelet" Sep 08 11:30:10.366571 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.366542 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 08 11:30:10.366674 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.366663 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 08 11:30:10.366899 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.366839 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 08 11:30:10.367344 ip-10-0-133-187 systemd[1]: Started Kubernetes Kubelet. Sep 08 11:30:10.368240 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.368221 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 08 11:30:10.369468 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.369454 2564 server.go:310] "Adding debug handlers to kubelet server" Sep 08 11:30:10.371401 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.371382 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 08 11:30:10.375182 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.375142 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 08 11:30:10.375343 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.374173 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-187.ec2.internal.18d355116f4e1e8d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-187.ec2.internal,UID:ip-10-0-133-187.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-187.ec2.internal,},FirstTimestamp:2026-09-08 11:30:10.366414477 +0000 UTC m=+0.359247431,LastTimestamp:2026-09-08 11:30:10.366414477 +0000 UTC m=+0.359247431,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-187.ec2.internal,}" Sep 08 11:30:10.376187 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.376053 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 08 11:30:10.376965 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.376403 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found" Sep 08 11:30:10.377090 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.377075 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 08 11:30:10.377157 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.377094 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 08 11:30:10.377351 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.377338 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 08 11:30:10.377601 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.377590 2564 reconstruct.go:97] "Volume reconstruction finished" Sep 08 11:30:10.377601 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.377601 2564 reconciler.go:29] "Reconciler: start to sync state" Sep 08 11:30:10.378040 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.377992 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 08 11:30:10.378040 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.378039 2564 factory.go:55] Registering systemd factory Sep 08 11:30:10.378143 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.378048 2564 factory.go:223] Registration of the systemd container factory successfully Sep 08 11:30:10.378357 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.378341 2564 factory.go:153] Registering CRI-O factory Sep 08 11:30:10.378401 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.378363 2564 factory.go:223] Registration of the crio container factory successfully Sep 08 11:30:10.378401 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.378390 2564 factory.go:103] Registering Raw factory Sep 08 11:30:10.378470 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.378404 2564 manager.go:1196] Started watching for new ooms in manager Sep 08 11:30:10.378833 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.378818 2564 manager.go:349] Starting recovery of all containers Sep 08 11:30:10.381937 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.381908 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 08 11:30:10.382032 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.382011 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 08 11:30:10.383229 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.383208 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ccmg8" Sep 08 11:30:10.389325 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.389204 2564 manager.go:354] Recovery completed Sep 08 11:30:10.394070 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.394056 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 11:30:10.394614 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.394599 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ccmg8" Sep 08 11:30:10.396008 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.395994 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientMemory" Sep 08 11:30:10.396061 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.396025 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 11:30:10.396061 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.396039 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientPID" Sep 08 11:30:10.396701 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.396684 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 08 11:30:10.396701 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.396700 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 08 11:30:10.396824 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.396718 2564 state_mem.go:36] "Initialized new in-memory state store" Sep 08 11:30:10.396824 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.396717 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 11:30:10.398041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.397932 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientMemory" Sep 08 11:30:10.398041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.397954 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 11:30:10.398041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.397971 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientPID" Sep 08 11:30:10.398361 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.398278 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-187.ec2.internal.18d355117111b524 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-187.ec2.internal,UID:ip-10-0-133-187.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-187.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-187.ec2.internal,},FirstTimestamp:2026-09-08 11:30:10.396009764 +0000 UTC m=+0.388842725,LastTimestamp:2026-09-08 11:30:10.396009764 +0000 UTC m=+0.388842725,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-187.ec2.internal,}" Sep 08 11:30:10.398803 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.398786 2564 policy_none.go:49] "None policy: Start" Sep 08 11:30:10.398869 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.398806 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 08 11:30:10.398869 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.398819 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 08 11:30:10.400875 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.400865 2564 policy_none.go:47] "Start" Sep 08 11:30:10.429351 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.429336 2564 manager.go:337] "Starting Device Plugin manager" Sep 08 11:30:10.429458 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.429373 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 08 11:30:10.429458 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.429383 2564 server.go:87] "Starting device plugin registration server" Sep 08 11:30:10.429918 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.429709 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 08 11:30:10.429918 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.429779 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 08 11:30:10.430256 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.429968 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 08 11:30:10.430256 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.430050 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 08 11:30:10.430256 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.430058 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 08 11:30:10.430757 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.430738 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 08 11:30:10.430828 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.430790 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-187.ec2.internal\" not found" Sep 08 11:30:10.511425 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.511397 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 08 11:30:10.512532 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.512517 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 08 11:30:10.512606 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.512582 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 08 11:30:10.512655 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.512610 2564 kubelet.go:2448] "Starting kubelet main sync loop" Sep 08 11:30:10.512714 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.512656 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 08 11:30:10.515162 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.515140 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 11:30:10.530140 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.530128 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 11:30:10.531210 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.531193 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientMemory" Sep 08 11:30:10.531295 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.531227 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 11:30:10.531295 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.531238 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientPID" Sep 08 11:30:10.531295 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.531258 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.543137 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.543055 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.543137 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.543087 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-187.ec2.internal\": node \"ip-10-0-133-187.ec2.internal\" not found" Sep 08 11:30:10.558973 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.558953 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found" Sep 08 11:30:10.613555 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.613512 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal"] Sep 08 11:30:10.613639 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.613612 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 11:30:10.615676 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.614621 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientMemory" Sep 08 11:30:10.615676 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.614654 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 11:30:10.615676 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.614667 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientPID" Sep 08 11:30:10.616971 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.616956 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 11:30:10.617101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.617075 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.617147 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.617115 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 11:30:10.617741 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.617711 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientMemory" Sep 08 11:30:10.617814 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.617746 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientMemory" Sep 08 11:30:10.617814 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.617771 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 11:30:10.617814 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.617781 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientPID" Sep 08 11:30:10.617900 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.617753 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 11:30:10.617900 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.617830 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientPID" Sep 08 11:30:10.619926 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.619913 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.619971 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.619936 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 11:30:10.620569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.620554 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientMemory" Sep 08 11:30:10.620635 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.620581 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 11:30:10.620635 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.620595 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeHasSufficientPID" Sep 08 11:30:10.643266 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.643250 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-187.ec2.internal\" not found" node="ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.647607 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.647592 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-187.ec2.internal\" not found" node="ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.659027 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.659007 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found" Sep 08 11:30:10.678504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.678486 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18abe9a9a9441b764e0c8f71a4e8bdc5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-187.ec2.internal\" (UID: \"18abe9a9a9441b764e0c8f71a4e8bdc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.678575 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.678510 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4035a6e430bc7a42c0e8baec502a110-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal\" (UID: \"b4035a6e430bc7a42c0e8baec502a110\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.678575 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.678529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4035a6e430bc7a42c0e8baec502a110-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal\" (UID: \"b4035a6e430bc7a42c0e8baec502a110\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.759832 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.759796 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found" Sep 08 11:30:10.779185 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.779159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4035a6e430bc7a42c0e8baec502a110-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal\" (UID: \"b4035a6e430bc7a42c0e8baec502a110\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.779269 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.779194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18abe9a9a9441b764e0c8f71a4e8bdc5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-187.ec2.internal\" (UID: \"18abe9a9a9441b764e0c8f71a4e8bdc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.779269 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.779230 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4035a6e430bc7a42c0e8baec502a110-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal\" (UID: \"b4035a6e430bc7a42c0e8baec502a110\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.779370 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.779282 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b4035a6e430bc7a42c0e8baec502a110-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal\" (UID: \"b4035a6e430bc7a42c0e8baec502a110\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.779370 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.779285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18abe9a9a9441b764e0c8f71a4e8bdc5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-187.ec2.internal\" (UID: \"18abe9a9a9441b764e0c8f71a4e8bdc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.779370 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.779290 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4035a6e430bc7a42c0e8baec502a110-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal\" (UID: \"b4035a6e430bc7a42c0e8baec502a110\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.860622 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.860547 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found" Sep 08 11:30:10.946107 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.946081 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.951043 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:10.950962 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal" Sep 08 11:30:10.960839 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:10.960821 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found" Sep 08 11:30:11.060940 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:11.060904 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found" Sep 08 11:30:11.161438 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:11.161372 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found" Sep 08 11:30:12.395629 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:12.395608 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-187.ec2.internal\" not found" Sep 08 11:30:12.416926 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.416904 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 11:30:12.444935 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.444915 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 08 11:30:12.445310 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.445069 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 11:30:12.445310 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.445085 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 11:30:12.489210 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.489193 2564 apiserver.go:52] "Watching apiserver" Sep 08 11:30:12.507142 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.507109 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 08 11:30:12.508270 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.508250 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-5vslm","openshift-image-registry/node-ca-sb576","openshift-multus/network-metrics-daemon-rr4sq","openshift-network-diagnostics/network-check-target-gbpbr","openshift-network-operator/iptables-alerter-zz9zq","kube-system/konnectivity-agent-bjthr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb","openshift-cluster-node-tuning-operator/tuned-wfswx","openshift-multus/multus-5dk5n","openshift-multus/multus-additional-cni-plugins-m6b84","openshift-ovn-kubernetes/ovnkube-node-5x6mb","kube-system/global-pull-secret-syncer-8r82d"] Sep 08 11:30:12.509144 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.509125 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 08 11:30:12.510125 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.510111 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal" Sep 08 11:30:12.510713 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.510695 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5vslm" Sep 08 11:30:12.512732 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.512716 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sb576" Sep 08 11:30:12.513468 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.513450 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tkm2n\"" Sep 08 11:30:12.513515 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.513490 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 08 11:30:12.513556 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.513492 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 08 11:30:12.514803 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.514784 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:12.514898 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:12.514880 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:12.515093 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.515076 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-876vj\" (UniqueName: \"kubernetes.io/projected/15e73439-0f50-4c2e-9155-3e18ec9e5231-kube-api-access-876vj\") pod \"node-resolver-5vslm\" (UID: \"15e73439-0f50-4c2e-9155-3e18ec9e5231\") " pod="openshift-dns/node-resolver-5vslm" Sep 08 11:30:12.515178 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.515100 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7djtj\" (UniqueName: \"kubernetes.io/projected/c6a08843-d61b-451e-8e36-9e949a01059d-kube-api-access-7djtj\") pod \"node-ca-sb576\" (UID: \"c6a08843-d61b-451e-8e36-9e949a01059d\") " pod="openshift-image-registry/node-ca-sb576" Sep 08 11:30:12.515178 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.515118 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/15e73439-0f50-4c2e-9155-3e18ec9e5231-hosts-file\") pod \"node-resolver-5vslm\" (UID: \"15e73439-0f50-4c2e-9155-3e18ec9e5231\") " pod="openshift-dns/node-resolver-5vslm" Sep 08 11:30:12.515178 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.515140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e73439-0f50-4c2e-9155-3e18ec9e5231-tmp-dir\") pod \"node-resolver-5vslm\" (UID: \"15e73439-0f50-4c2e-9155-3e18ec9e5231\") " pod="openshift-dns/node-resolver-5vslm" Sep 08 11:30:12.515178 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.515162 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6a08843-d61b-451e-8e36-9e949a01059d-host\") pod \"node-ca-sb576\" (UID: \"c6a08843-d61b-451e-8e36-9e949a01059d\") " pod="openshift-image-registry/node-ca-sb576" Sep 08 11:30:12.515475 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.515187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6a08843-d61b-451e-8e36-9e949a01059d-serviceca\") pod \"node-ca-sb576\" (UID: \"c6a08843-d61b-451e-8e36-9e949a01059d\") " pod="openshift-image-registry/node-ca-sb576" Sep 08 11:30:12.515475 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.515211 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 08 11:30:12.515475 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.515248 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 08 11:30:12.515475 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.515263 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 08 11:30:12.515656 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.515554 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8lvj4\"" Sep 08 11:30:12.516840 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.516826 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:12.516886 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:12.516872 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:12.518692 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.518679 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 11:30:12.518742 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.518732 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" Sep 08 11:30:12.518961 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.518942 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zz9zq" Sep 08 11:30:12.521028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.521012 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bjthr" Sep 08 11:30:12.521356 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.521336 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5f2gd\"" Sep 08 11:30:12.521451 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.521392 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 08 11:30:12.521508 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.521450 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 08 11:30:12.521508 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.521476 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 08 11:30:12.521508 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.521492 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 11:30:12.523228 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.523215 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.523414 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.523337 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wcm6b\"" Sep 08 11:30:12.523996 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.523340 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 08 11:30:12.524096 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.524004 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 08 11:30:12.526045 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.526005 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 11:30:12.526284 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.526264 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.526404 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.526331 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 08 11:30:12.526781 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.526761 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 08 11:30:12.526781 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.526777 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 08 11:30:12.526888 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.526765 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fdzwn\"" Sep 08 11:30:12.528960 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.528943 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.530002 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.529974 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 08 11:30:12.530094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.530066 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jsb4t\"" Sep 08 11:30:12.530147 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.530133 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 11:30:12.530256 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.530240 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 08 11:30:12.530293 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.530282 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 11:30:12.531079 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.531036 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 11:25:11 +0000 UTC" deadline="2028-05-29 17:03:51.925259508 +0000 UTC" Sep 08 11:30:12.531135 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.531078 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15101h33m39.394185156s" Sep 08 11:30:12.531191 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.531175 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 08 11:30:12.531243 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.531180 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 08 11:30:12.531243 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.531225 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-md8wx\"" Sep 08 11:30:12.531328 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.531261 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 08 11:30:12.531362 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.531330 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.531490 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.531474 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 08 11:30:12.533423 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.533408 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 08 11:30:12.533650 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.533636 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.533650 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.533644 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qkrr8\"" Sep 08 11:30:12.533773 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.533673 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 08 11:30:12.535975 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.535952 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal"] Sep 08 11:30:12.535975 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.535971 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hjb96\"" Sep 08 11:30:12.536077 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.535992 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal"] Sep 08 11:30:12.536077 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.536071 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:12.536138 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.536094 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 08 11:30:12.536138 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:12.536121 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:12.537100 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.537084 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 08 11:30:12.537176 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.537086 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 08 11:30:12.537176 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.537128 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 08 11:30:12.537266 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.537085 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 08 11:30:12.537266 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.537085 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 08 11:30:12.552517 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.552501 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kh5jj" Sep 08 11:30:12.562850 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.562833 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kh5jj" Sep 08 11:30:12.611934 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.611918 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.615819 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-sys-fs\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.615860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6a08843-d61b-451e-8e36-9e949a01059d-host\") pod \"node-ca-sb576\" (UID: \"c6a08843-d61b-451e-8e36-9e949a01059d\") " pod="openshift-image-registry/node-ca-sb576" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.615895 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zqg9\" (UniqueName: \"kubernetes.io/projected/d96c6817-e9cc-47af-a268-2516ca461639-kube-api-access-5zqg9\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.615926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-run-k8s-cni-cncf-io\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.615956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74934df5-b10d-497b-9ada-49aae488dd2b-dbus\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.615981 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3eb89e07-e389-4c2f-9679-5867eecb4b2e-cni-binary-copy\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djtj\" (UniqueName: \"kubernetes.io/projected/c6a08843-d61b-451e-8e36-9e949a01059d-kube-api-access-7djtj\") pod \"node-ca-sb576\" (UID: \"c6a08843-d61b-451e-8e36-9e949a01059d\") " pod="openshift-image-registry/node-ca-sb576" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmbjj\" (UniqueName: \"kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj\") pod \"network-check-target-gbpbr\" (UID: \"16e669b2-9d2f-45ba-bb6a-447a8e87ae85\") " pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616118 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-modprobe-d\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616149 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-slash\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-run-openvswitch\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82d66\" (UniqueName: \"kubernetes.io/projected/b03e1094-57b4-44d3-828f-f77b4a65a2b9-kube-api-access-82d66\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616259 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb89e07-e389-4c2f-9679-5867eecb4b2e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42627895-7836-4cee-bb30-53fad710041e-cni-binary-copy\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616316 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-hostroot\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.617041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616340 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-etc-openvswitch\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616569 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6a08843-d61b-451e-8e36-9e949a01059d-host\") pod \"node-ca-sb576\" (UID: \"c6a08843-d61b-451e-8e36-9e949a01059d\") " pod="openshift-image-registry/node-ca-sb576" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616637 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/625b8b89-b76a-4075-901d-42ec3e6cb070-env-overrides\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616669 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bec11709-37ed-4639-ae01-67ee4c64a5f7-iptables-alerter-script\") pod \"iptables-alerter-zz9zq\" (UID: \"bec11709-37ed-4639-ae01-67ee4c64a5f7\") " pod="openshift-network-operator/iptables-alerter-zz9zq" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-876vj\" (UniqueName: \"kubernetes.io/projected/15e73439-0f50-4c2e-9155-3e18ec9e5231-kube-api-access-876vj\") pod \"node-resolver-5vslm\" (UID: \"15e73439-0f50-4c2e-9155-3e18ec9e5231\") " pod="openshift-dns/node-resolver-5vslm" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616741 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-run-ovn\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/625b8b89-b76a-4075-901d-42ec3e6cb070-ovnkube-script-lib\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616801 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3eb89e07-e389-4c2f-9679-5867eecb4b2e-system-cni-dir\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616842 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-sysctl-conf\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616868 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-run-netns\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616891 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-systemd-units\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616918 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-run-netns\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.616974 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/625b8b89-b76a-4075-901d-42ec3e6cb070-ovn-node-metrics-cert\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.617013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs\") pod \"network-metrics-daemon-rr4sq\" (UID: \"e3b04def-aa09-4e3b-ba69-8a8838695a83\") " pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.617037 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-sys\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.617065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-run-systemd\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.617125 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-cni-netd\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.618247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.617182 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6a08843-d61b-451e-8e36-9e949a01059d-serviceca\") pod \"node-ca-sb576\" (UID: \"c6a08843-d61b-451e-8e36-9e949a01059d\") " pod="openshift-image-registry/node-ca-sb576" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.617222 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-sysconfig\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.617717 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6a08843-d61b-451e-8e36-9e949a01059d-serviceca\") pod \"node-ca-sb576\" (UID: \"c6a08843-d61b-451e-8e36-9e949a01059d\") " pod="openshift-image-registry/node-ca-sb576" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618149 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-systemd\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618181 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-cnibin\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618204 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmm8f\" (UniqueName: \"kubernetes.io/projected/42627895-7836-4cee-bb30-53fad710041e-kube-api-access-wmm8f\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618222 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-log-socket\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618275 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-socket-dir\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618310 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-var-lib-kubelet\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4vw5\" (UniqueName: \"kubernetes.io/projected/e3b04def-aa09-4e3b-ba69-8a8838695a83-kube-api-access-n4vw5\") pod \"network-metrics-daemon-rr4sq\" (UID: \"e3b04def-aa09-4e3b-ba69-8a8838695a83\") " pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618377 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-registration-dir\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618403 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-device-dir\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618434 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e73439-0f50-4c2e-9155-3e18ec9e5231-tmp-dir\") pod \"node-resolver-5vslm\" (UID: \"15e73439-0f50-4c2e-9155-3e18ec9e5231\") " pod="openshift-dns/node-resolver-5vslm" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618492 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-multus-cni-dir\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-var-lib-cni-bin\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618865 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-kubernetes\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.618897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d96c6817-e9cc-47af-a268-2516ca461639-etc-tuned\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618904 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-run-multus-certs\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74934df5-b10d-497b-9ada-49aae488dd2b-kubelet-config\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618943 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-767d7\" (UniqueName: \"kubernetes.io/projected/bec11709-37ed-4639-ae01-67ee4c64a5f7-kube-api-access-767d7\") pod \"iptables-alerter-zz9zq\" (UID: \"bec11709-37ed-4639-ae01-67ee4c64a5f7\") " pod="openshift-network-operator/iptables-alerter-zz9zq" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618959 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3eb89e07-e389-4c2f-9679-5867eecb4b2e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.618998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-sysctl-d\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619015 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e73439-0f50-4c2e-9155-3e18ec9e5231-tmp-dir\") pod \"node-resolver-5vslm\" (UID: \"15e73439-0f50-4c2e-9155-3e18ec9e5231\") " pod="openshift-dns/node-resolver-5vslm" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619025 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/42627895-7836-4cee-bb30-53fad710041e-multus-daemon-config\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619080 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-etc-kubernetes\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-kubelet\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3eb89e07-e389-4c2f-9679-5867eecb4b2e-cnibin\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619154 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/445df2c7-2335-4a07-bf1d-ad753e708cc3-konnectivity-ca\") pod \"konnectivity-agent-bjthr\" (UID: \"445df2c7-2335-4a07-bf1d-ad753e708cc3\") " pod="kube-system/konnectivity-agent-bjthr" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619176 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-run\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619192 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-var-lib-openvswitch\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619212 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/15e73439-0f50-4c2e-9155-3e18ec9e5231-hosts-file\") pod \"node-resolver-5vslm\" (UID: \"15e73439-0f50-4c2e-9155-3e18ec9e5231\") " pod="openshift-dns/node-resolver-5vslm" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619269 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/15e73439-0f50-4c2e-9155-3e18ec9e5231-hosts-file\") pod \"node-resolver-5vslm\" (UID: \"15e73439-0f50-4c2e-9155-3e18ec9e5231\") " pod="openshift-dns/node-resolver-5vslm" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-host\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.619592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619307 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-var-lib-cni-multus\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619320 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-multus-conf-dir\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619335 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-lib-modules\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d96c6817-e9cc-47af-a268-2516ca461639-tmp\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619362 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-system-cni-dir\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619393 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-multus-socket-dir-parent\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-var-lib-kubelet\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619427 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-cni-bin\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619441 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619457 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/445df2c7-2335-4a07-bf1d-ad753e708cc3-agent-certs\") pod \"konnectivity-agent-bjthr\" (UID: \"445df2c7-2335-4a07-bf1d-ad753e708cc3\") " pod="kube-system/konnectivity-agent-bjthr" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619480 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/625b8b89-b76a-4075-901d-42ec3e6cb070-ovnkube-config\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619505 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bec11709-37ed-4639-ae01-67ee4c64a5f7-host-slash\") pod \"iptables-alerter-zz9zq\" (UID: \"bec11709-37ed-4639-ae01-67ee4c64a5f7\") " pod="openshift-network-operator/iptables-alerter-zz9zq" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619524 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3eb89e07-e389-4c2f-9679-5867eecb4b2e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phqt2\" (UniqueName: \"kubernetes.io/projected/3eb89e07-e389-4c2f-9679-5867eecb4b2e-kube-api-access-phqt2\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-node-log\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.620124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619603 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-run-ovn-kubernetes\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.620602 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619622 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbchx\" (UniqueName: \"kubernetes.io/projected/625b8b89-b76a-4075-901d-42ec3e6cb070-kube-api-access-nbchx\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.620602 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.619644 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.625534 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.625513 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 08 11:30:12.628581 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.628557 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-876vj\" (UniqueName: \"kubernetes.io/projected/15e73439-0f50-4c2e-9155-3e18ec9e5231-kube-api-access-876vj\") pod \"node-resolver-5vslm\" (UID: \"15e73439-0f50-4c2e-9155-3e18ec9e5231\") " pod="openshift-dns/node-resolver-5vslm" Sep 08 11:30:12.628785 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.628769 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7djtj\" (UniqueName: \"kubernetes.io/projected/c6a08843-d61b-451e-8e36-9e949a01059d-kube-api-access-7djtj\") pod \"node-ca-sb576\" (UID: \"c6a08843-d61b-451e-8e36-9e949a01059d\") " pod="openshift-image-registry/node-ca-sb576" Sep 08 11:30:12.720718 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.720687 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42627895-7836-4cee-bb30-53fad710041e-cni-binary-copy\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.720718 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.720717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-hostroot\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.720970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.720733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-etc-openvswitch\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.720970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.720750 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/625b8b89-b76a-4075-901d-42ec3e6cb070-env-overrides\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.720970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.720765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bec11709-37ed-4639-ae01-67ee4c64a5f7-iptables-alerter-script\") pod \"iptables-alerter-zz9zq\" (UID: \"bec11709-37ed-4639-ae01-67ee4c64a5f7\") " pod="openshift-network-operator/iptables-alerter-zz9zq" Sep 08 11:30:12.720970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.720805 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-hostroot\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.720970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.720815 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-etc-openvswitch\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.720970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.720811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-run-ovn\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.720970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.720856 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/625b8b89-b76a-4075-901d-42ec3e6cb070-ovnkube-script-lib\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.720970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.720924 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-run-ovn\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.720970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.720930 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3eb89e07-e389-4c2f-9679-5867eecb4b2e-system-cni-dir\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.720970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.720953 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3eb89e07-e389-4c2f-9679-5867eecb4b2e-system-cni-dir\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.720970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.720969 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-sysctl-conf\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.720992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-run-netns\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-systemd-units\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721039 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-run-netns\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721056 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-run-netns\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721084 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/625b8b89-b76a-4075-901d-42ec3e6cb070-ovn-node-metrics-cert\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721089 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-systemd-units\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs\") pod \"network-metrics-daemon-rr4sq\" (UID: \"e3b04def-aa09-4e3b-ba69-8a8838695a83\") " pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-run-netns\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721130 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-sys\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721152 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-run-systemd\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721157 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-sysctl-conf\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-cni-netd\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721207 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-sys\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721229 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-sysconfig\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-systemd\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721275 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-cnibin\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:12.721285 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 11:30:12.721516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmm8f\" (UniqueName: \"kubernetes.io/projected/42627895-7836-4cee-bb30-53fad710041e-kube-api-access-wmm8f\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721318 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-cni-netd\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721322 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-log-socket\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721363 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-run-systemd\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:12.721396 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs podName:e3b04def-aa09-4e3b-ba69-8a8838695a83 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:13.221339289 +0000 UTC m=+2.080526597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs") pod "network-metrics-daemon-rr4sq" (UID: "e3b04def-aa09-4e3b-ba69-8a8838695a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/625b8b89-b76a-4075-901d-42ec3e6cb070-env-overrides\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721435 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-log-socket\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721443 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bec11709-37ed-4639-ae01-67ee4c64a5f7-iptables-alerter-script\") pod \"iptables-alerter-zz9zq\" (UID: \"bec11709-37ed-4639-ae01-67ee4c64a5f7\") " pod="openshift-network-operator/iptables-alerter-zz9zq" Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721466 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-systemd\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721473 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721482 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/625b8b89-b76a-4075-901d-42ec3e6cb070-ovnkube-script-lib\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721519 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-socket-dir\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721524 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-cnibin\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721534 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-sysconfig\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721547 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-var-lib-kubelet\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721572 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4vw5\" (UniqueName: \"kubernetes.io/projected/e3b04def-aa09-4e3b-ba69-8a8838695a83-kube-api-access-n4vw5\") pod \"network-metrics-daemon-rr4sq\" (UID: \"e3b04def-aa09-4e3b-ba69-8a8838695a83\") " pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:12.721591 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 11:30:12.722301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721596 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-registration-dir\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-device-dir\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721628 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-socket-dir\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721634 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-var-lib-kubelet\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:12.721639 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret podName:74934df5-b10d-497b-9ada-49aae488dd2b nodeName:}" failed. No retries permitted until 2026-09-08 11:30:13.221622582 +0000 UTC m=+2.080809881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret") pod "global-pull-secret-syncer-8r82d" (UID: "74934df5-b10d-497b-9ada-49aae488dd2b") : object "kube-system"/"original-pull-secret" not registered Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721665 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-device-dir\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-multus-cni-dir\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-var-lib-cni-bin\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721734 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-multus-cni-dir\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721745 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-kubernetes\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721693 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-registration-dir\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d96c6817-e9cc-47af-a268-2516ca461639-etc-tuned\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-run-multus-certs\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721823 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74934df5-b10d-497b-9ada-49aae488dd2b-kubelet-config\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721826 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-kubernetes\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721797 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-var-lib-cni-bin\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721836 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-run-multus-certs\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.723169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721842 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42627895-7836-4cee-bb30-53fad710041e-cni-binary-copy\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721867 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-767d7\" (UniqueName: \"kubernetes.io/projected/bec11709-37ed-4639-ae01-67ee4c64a5f7-kube-api-access-767d7\") pod \"iptables-alerter-zz9zq\" (UID: \"bec11709-37ed-4639-ae01-67ee4c64a5f7\") " pod="openshift-network-operator/iptables-alerter-zz9zq" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721873 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74934df5-b10d-497b-9ada-49aae488dd2b-kubelet-config\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721895 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3eb89e07-e389-4c2f-9679-5867eecb4b2e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721918 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-sysctl-d\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721938 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/42627895-7836-4cee-bb30-53fad710041e-multus-daemon-config\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.721960 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-etc-kubernetes\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722026 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-etc-kubernetes\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-kubelet\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722057 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-sysctl-d\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3eb89e07-e389-4c2f-9679-5867eecb4b2e-cnibin\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/445df2c7-2335-4a07-bf1d-ad753e708cc3-konnectivity-ca\") pod \"konnectivity-agent-bjthr\" (UID: \"445df2c7-2335-4a07-bf1d-ad753e708cc3\") " pod="kube-system/konnectivity-agent-bjthr" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722158 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-run\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722169 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3eb89e07-e389-4c2f-9679-5867eecb4b2e-cnibin\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722131 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-kubelet\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722187 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-var-lib-openvswitch\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722223 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-host\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-var-lib-cni-multus\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.724028 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722263 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-run\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722223 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-var-lib-openvswitch\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722280 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-multus-conf-dir\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-lib-modules\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722315 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-host\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722327 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d96c6817-e9cc-47af-a268-2516ca461639-tmp\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722350 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-multus-conf-dir\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-system-cni-dir\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722404 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-multus-socket-dir-parent\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-var-lib-kubelet\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722413 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-system-cni-dir\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-cni-bin\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722479 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/42627895-7836-4cee-bb30-53fad710041e-multus-daemon-config\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722493 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-multus-socket-dir-parent\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722448 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3eb89e07-e389-4c2f-9679-5867eecb4b2e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722519 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-cni-bin\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722553 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-lib-modules\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.724846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722557 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/445df2c7-2335-4a07-bf1d-ad753e708cc3-agent-certs\") pod \"konnectivity-agent-bjthr\" (UID: \"445df2c7-2335-4a07-bf1d-ad753e708cc3\") " pod="kube-system/konnectivity-agent-bjthr" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722594 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-var-lib-kubelet\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722680 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-var-lib-cni-multus\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722682 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/625b8b89-b76a-4075-901d-42ec3e6cb070-ovnkube-config\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722720 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/445df2c7-2335-4a07-bf1d-ad753e708cc3-konnectivity-ca\") pod \"konnectivity-agent-bjthr\" (UID: \"445df2c7-2335-4a07-bf1d-ad753e708cc3\") " pod="kube-system/konnectivity-agent-bjthr" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722722 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bec11709-37ed-4639-ae01-67ee4c64a5f7-host-slash\") pod \"iptables-alerter-zz9zq\" (UID: \"bec11709-37ed-4639-ae01-67ee4c64a5f7\") " pod="openshift-network-operator/iptables-alerter-zz9zq" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722753 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bec11709-37ed-4639-ae01-67ee4c64a5f7-host-slash\") pod \"iptables-alerter-zz9zq\" (UID: \"bec11709-37ed-4639-ae01-67ee4c64a5f7\") " pod="openshift-network-operator/iptables-alerter-zz9zq" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722779 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3eb89e07-e389-4c2f-9679-5867eecb4b2e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phqt2\" (UniqueName: \"kubernetes.io/projected/3eb89e07-e389-4c2f-9679-5867eecb4b2e-kube-api-access-phqt2\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-node-log\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-run-ovn-kubernetes\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbchx\" (UniqueName: \"kubernetes.io/projected/625b8b89-b76a-4075-901d-42ec3e6cb070-kube-api-access-nbchx\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722950 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722975 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-sys-fs\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zqg9\" (UniqueName: \"kubernetes.io/projected/d96c6817-e9cc-47af-a268-2516ca461639-kube-api-access-5zqg9\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.725346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723026 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-run-k8s-cni-cncf-io\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723055 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74934df5-b10d-497b-9ada-49aae488dd2b-dbus\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3eb89e07-e389-4c2f-9679-5867eecb4b2e-cni-binary-copy\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723104 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbjj\" (UniqueName: \"kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj\") pod \"network-check-target-gbpbr\" (UID: \"16e669b2-9d2f-45ba-bb6a-447a8e87ae85\") " pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723127 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-modprobe-d\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-slash\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-run-openvswitch\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82d66\" (UniqueName: \"kubernetes.io/projected/b03e1094-57b4-44d3-828f-f77b4a65a2b9-kube-api-access-82d66\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb89e07-e389-4c2f-9679-5867eecb4b2e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723246 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3eb89e07-e389-4c2f-9679-5867eecb4b2e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723300 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-node-log\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723306 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/625b8b89-b76a-4075-901d-42ec3e6cb070-ovnkube-config\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723335 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-run-ovn-kubernetes\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.722906 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723435 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-etc-selinux\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723482 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b03e1094-57b4-44d3-828f-f77b4a65a2b9-sys-fs\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723637 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d96c6817-e9cc-47af-a268-2516ca461639-etc-modprobe-d\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.725838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723695 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74934df5-b10d-497b-9ada-49aae488dd2b-dbus\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:12.726300 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723727 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb89e07-e389-4c2f-9679-5867eecb4b2e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.726300 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/42627895-7836-4cee-bb30-53fad710041e-host-run-k8s-cni-cncf-io\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.726300 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723830 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-host-slash\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.726300 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.723869 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/625b8b89-b76a-4075-901d-42ec3e6cb070-run-openvswitch\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.726300 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.724152 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3eb89e07-e389-4c2f-9679-5867eecb4b2e-cni-binary-copy\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.726300 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.724354 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d96c6817-e9cc-47af-a268-2516ca461639-etc-tuned\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.726300 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.724501 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/625b8b89-b76a-4075-901d-42ec3e6cb070-ovn-node-metrics-cert\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.726300 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.724663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d96c6817-e9cc-47af-a268-2516ca461639-tmp\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.726300 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.724971 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/445df2c7-2335-4a07-bf1d-ad753e708cc3-agent-certs\") pod \"konnectivity-agent-bjthr\" (UID: \"445df2c7-2335-4a07-bf1d-ad753e708cc3\") " pod="kube-system/konnectivity-agent-bjthr" Sep 08 11:30:12.730271 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.730248 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmm8f\" (UniqueName: \"kubernetes.io/projected/42627895-7836-4cee-bb30-53fad710041e-kube-api-access-wmm8f\") pod \"multus-5dk5n\" (UID: \"42627895-7836-4cee-bb30-53fad710041e\") " pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.733401 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:12.733348 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 11:30:12.733496 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:12.733407 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 11:30:12.733496 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:12.733420 2564 projected.go:196] Error preparing data for projected volume kube-api-access-hmbjj for pod openshift-network-diagnostics/network-check-target-gbpbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 11:30:12.733496 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:12.733473 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj podName:16e669b2-9d2f-45ba-bb6a-447a8e87ae85 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:13.233456987 +0000 UTC m=+2.092644297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hmbjj" (UniqueName: "kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj") pod "network-check-target-gbpbr" (UID: "16e669b2-9d2f-45ba-bb6a-447a8e87ae85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 11:30:12.735009 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.734980 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-767d7\" (UniqueName: \"kubernetes.io/projected/bec11709-37ed-4639-ae01-67ee4c64a5f7-kube-api-access-767d7\") pod \"iptables-alerter-zz9zq\" (UID: \"bec11709-37ed-4639-ae01-67ee4c64a5f7\") " pod="openshift-network-operator/iptables-alerter-zz9zq" Sep 08 11:30:12.735503 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.735482 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phqt2\" (UniqueName: \"kubernetes.io/projected/3eb89e07-e389-4c2f-9679-5867eecb4b2e-kube-api-access-phqt2\") pod \"multus-additional-cni-plugins-m6b84\" (UID: \"3eb89e07-e389-4c2f-9679-5867eecb4b2e\") " pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.735862 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.735829 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zqg9\" (UniqueName: \"kubernetes.io/projected/d96c6817-e9cc-47af-a268-2516ca461639-kube-api-access-5zqg9\") pod \"tuned-wfswx\" (UID: \"d96c6817-e9cc-47af-a268-2516ca461639\") " pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.735966 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.735947 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4vw5\" (UniqueName: \"kubernetes.io/projected/e3b04def-aa09-4e3b-ba69-8a8838695a83-kube-api-access-n4vw5\") pod \"network-metrics-daemon-rr4sq\" (UID: \"e3b04def-aa09-4e3b-ba69-8a8838695a83\") " pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:12.736025 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.735976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbchx\" (UniqueName: \"kubernetes.io/projected/625b8b89-b76a-4075-901d-42ec3e6cb070-kube-api-access-nbchx\") pod \"ovnkube-node-5x6mb\" (UID: \"625b8b89-b76a-4075-901d-42ec3e6cb070\") " pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.736491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.736476 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82d66\" (UniqueName: \"kubernetes.io/projected/b03e1094-57b4-44d3-828f-f77b4a65a2b9-kube-api-access-82d66\") pod \"aws-ebs-csi-driver-node-pwzxb\" (UID: \"b03e1094-57b4-44d3-828f-f77b4a65a2b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.820793 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:12.820644 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4035a6e430bc7a42c0e8baec502a110.slice/crio-2c499a5f5c080f01bb16484edca776d96807fff80064e07ac764d37be1f4837d WatchSource:0}: Error finding container 2c499a5f5c080f01bb16484edca776d96807fff80064e07ac764d37be1f4837d: Status 404 returned error can't find the container with id 2c499a5f5c080f01bb16484edca776d96807fff80064e07ac764d37be1f4837d Sep 08 11:30:12.820982 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:12.820963 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18abe9a9a9441b764e0c8f71a4e8bdc5.slice/crio-b165df3d560a6cbbb38b570edc5edd132b4e0c95add1f9071d7b9f58ad87c52c WatchSource:0}: Error finding container b165df3d560a6cbbb38b570edc5edd132b4e0c95add1f9071d7b9f58ad87c52c: Status 404 returned error can't find the container with id b165df3d560a6cbbb38b570edc5edd132b4e0c95add1f9071d7b9f58ad87c52c Sep 08 11:30:12.825059 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.825044 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 11:30:12.838197 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.838179 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5vslm" Sep 08 11:30:12.847880 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:12.847856 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e73439_0f50_4c2e_9155_3e18ec9e5231.slice/crio-d0ac85f91c0c626154df0ba2b0746b63613a3758ecba4c9a179e965ca134dfea WatchSource:0}: Error finding container d0ac85f91c0c626154df0ba2b0746b63613a3758ecba4c9a179e965ca134dfea: Status 404 returned error can't find the container with id d0ac85f91c0c626154df0ba2b0746b63613a3758ecba4c9a179e965ca134dfea Sep 08 11:30:12.861480 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.861460 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sb576" Sep 08 11:30:12.871315 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:12.871289 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a08843_d61b_451e_8e36_9e949a01059d.slice/crio-c9111336381d595441d4e092f2f1f1101a3e48094d4f531939bc46914897f159 WatchSource:0}: Error finding container c9111336381d595441d4e092f2f1f1101a3e48094d4f531939bc46914897f159: Status 404 returned error can't find the container with id c9111336381d595441d4e092f2f1f1101a3e48094d4f531939bc46914897f159 Sep 08 11:30:12.878161 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.878145 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zz9zq" Sep 08 11:30:12.885336 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:12.885316 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec11709_37ed_4639_ae01_67ee4c64a5f7.slice/crio-5eaecc0725306d81ad9938625e90532c35d6aca4dad2c6927448cfdbd9a7bb0b WatchSource:0}: Error finding container 5eaecc0725306d81ad9938625e90532c35d6aca4dad2c6927448cfdbd9a7bb0b: Status 404 returned error can't find the container with id 5eaecc0725306d81ad9938625e90532c35d6aca4dad2c6927448cfdbd9a7bb0b Sep 08 11:30:12.893001 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.892983 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bjthr" Sep 08 11:30:12.900056 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:12.900037 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445df2c7_2335_4a07_bf1d_ad753e708cc3.slice/crio-c9ad916a2a774fa8ff4a84ecd5d3a45ff2393d81a9efba82a6e5ae5a88055b6f WatchSource:0}: Error finding container c9ad916a2a774fa8ff4a84ecd5d3a45ff2393d81a9efba82a6e5ae5a88055b6f: Status 404 returned error can't find the container with id c9ad916a2a774fa8ff4a84ecd5d3a45ff2393d81a9efba82a6e5ae5a88055b6f Sep 08 11:30:12.906605 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.906588 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" Sep 08 11:30:12.909431 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.909296 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wfswx" Sep 08 11:30:12.916217 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:12.916193 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03e1094_57b4_44d3_828f_f77b4a65a2b9.slice/crio-5c5c81f5739acf370752c1bf9d637da8aea4a968db853bf2be3c85edcca3f963 WatchSource:0}: Error finding container 5c5c81f5739acf370752c1bf9d637da8aea4a968db853bf2be3c85edcca3f963: Status 404 returned error can't find the container with id 5c5c81f5739acf370752c1bf9d637da8aea4a968db853bf2be3c85edcca3f963 Sep 08 11:30:12.919528 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:12.919506 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96c6817_e9cc_47af_a268_2516ca461639.slice/crio-31989b514d4b4a01b7f629e18db1a75ea07d19713abc5297510a5fa49e165970 WatchSource:0}: Error finding container 31989b514d4b4a01b7f629e18db1a75ea07d19713abc5297510a5fa49e165970: Status 404 returned error can't find the container with id 31989b514d4b4a01b7f629e18db1a75ea07d19713abc5297510a5fa49e165970 Sep 08 11:30:12.929124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.929107 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5dk5n" Sep 08 11:30:12.938361 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:12.938342 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42627895_7836_4cee_bb30_53fad710041e.slice/crio-580490baf615419367f88ebc4e463a9a8d41b11d2aed3950884ad9cbb5082b34 WatchSource:0}: Error finding container 580490baf615419367f88ebc4e463a9a8d41b11d2aed3950884ad9cbb5082b34: Status 404 returned error can't find the container with id 580490baf615419367f88ebc4e463a9a8d41b11d2aed3950884ad9cbb5082b34 Sep 08 11:30:12.950775 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.950757 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m6b84" Sep 08 11:30:12.953619 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:12.953567 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:12.959666 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:12.959598 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb89e07_e389_4c2f_9679_5867eecb4b2e.slice/crio-ba3c326e2c01c1fb42716c146b4662c72fc362f06448e3383209f958e9d7f1d5 WatchSource:0}: Error finding container ba3c326e2c01c1fb42716c146b4662c72fc362f06448e3383209f958e9d7f1d5: Status 404 returned error can't find the container with id ba3c326e2c01c1fb42716c146b4662c72fc362f06448e3383209f958e9d7f1d5 Sep 08 11:30:12.962535 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:12.962464 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625b8b89_b76a_4075_901d_42ec3e6cb070.slice/crio-b88a37e2305ada454ab96e32428d705135a942c4b78bf4e0c05fd91238115750 WatchSource:0}: Error finding container b88a37e2305ada454ab96e32428d705135a942c4b78bf4e0c05fd91238115750: Status 404 returned error can't find the container with id b88a37e2305ada454ab96e32428d705135a942c4b78bf4e0c05fd91238115750 Sep 08 11:30:12.964327 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:12.964308 2564 kuberuntime_manager.go:1459] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-5x6mb_openshift-ovn-kubernetes(625b8b89-b76a-4075-901d-42ec3e6cb070): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Sep 08 11:30:12.965679 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:12.965665 2564 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-5x6mb_openshift-ovn-kubernetes(625b8b89-b76a-4075-901d-42ec3e6cb070): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Sep 08 11:30:12.968649 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:12.968612 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" podUID="625b8b89-b76a-4075-901d-42ec3e6cb070" Sep 08 11:30:13.226710 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.226632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:13.226710 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.226690 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs\") pod \"network-metrics-daemon-rr4sq\" (UID: \"e3b04def-aa09-4e3b-ba69-8a8838695a83\") " pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:13.226879 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:13.226764 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 11:30:13.226879 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:13.226765 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 11:30:13.226879 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:13.226810 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs podName:e3b04def-aa09-4e3b-ba69-8a8838695a83 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:14.226798061 +0000 UTC m=+3.085985356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs") pod "network-metrics-daemon-rr4sq" (UID: "e3b04def-aa09-4e3b-ba69-8a8838695a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 11:30:13.226879 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:13.226824 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret podName:74934df5-b10d-497b-9ada-49aae488dd2b nodeName:}" failed. No retries permitted until 2026-09-08 11:30:14.226817157 +0000 UTC m=+3.086004452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret") pod "global-pull-secret-syncer-8r82d" (UID: "74934df5-b10d-497b-9ada-49aae488dd2b") : object "kube-system"/"original-pull-secret" not registered Sep 08 11:30:13.327414 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.327379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbjj\" (UniqueName: \"kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj\") pod \"network-check-target-gbpbr\" (UID: \"16e669b2-9d2f-45ba-bb6a-447a8e87ae85\") " pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:13.327585 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:13.327483 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 11:30:13.327585 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:13.327504 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 11:30:13.327585 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:13.327513 2564 projected.go:196] Error preparing data for projected volume kube-api-access-hmbjj for pod openshift-network-diagnostics/network-check-target-gbpbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 11:30:13.327585 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:13.327566 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj podName:16e669b2-9d2f-45ba-bb6a-447a8e87ae85 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:14.32754824 +0000 UTC m=+3.186735542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmbjj" (UniqueName: "kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj") pod "network-check-target-gbpbr" (UID: "16e669b2-9d2f-45ba-bb6a-447a8e87ae85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 11:30:13.565549 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.565390 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 11:25:12 +0000 UTC" deadline="2028-04-16 23:07:12.659817921 +0000 UTC" Sep 08 11:30:13.565549 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.565447 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14075h36m59.094375308s" Sep 08 11:30:13.649819 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.649788 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:13.649983 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:13.649917 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:13.654857 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.654804 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6b84" event={"ID":"3eb89e07-e389-4c2f-9679-5867eecb4b2e","Type":"ContainerStarted","Data":"ba3c326e2c01c1fb42716c146b4662c72fc362f06448e3383209f958e9d7f1d5"} Sep 08 11:30:13.657470 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.657443 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5dk5n" event={"ID":"42627895-7836-4cee-bb30-53fad710041e","Type":"ContainerStarted","Data":"580490baf615419367f88ebc4e463a9a8d41b11d2aed3950884ad9cbb5082b34"} Sep 08 11:30:13.659975 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.659952 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wfswx" event={"ID":"d96c6817-e9cc-47af-a268-2516ca461639","Type":"ContainerStarted","Data":"31989b514d4b4a01b7f629e18db1a75ea07d19713abc5297510a5fa49e165970"} Sep 08 11:30:13.664027 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.664001 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5vslm" event={"ID":"15e73439-0f50-4c2e-9155-3e18ec9e5231","Type":"ContainerStarted","Data":"d0ac85f91c0c626154df0ba2b0746b63613a3758ecba4c9a179e965ca134dfea"} Sep 08 11:30:13.666676 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.666652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal" event={"ID":"18abe9a9a9441b764e0c8f71a4e8bdc5","Type":"ContainerStarted","Data":"b165df3d560a6cbbb38b570edc5edd132b4e0c95add1f9071d7b9f58ad87c52c"} Sep 08 11:30:13.668124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.668102 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" event={"ID":"b4035a6e430bc7a42c0e8baec502a110","Type":"ContainerStarted","Data":"2c499a5f5c080f01bb16484edca776d96807fff80064e07ac764d37be1f4837d"} Sep 08 11:30:13.671468 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.671444 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" event={"ID":"625b8b89-b76a-4075-901d-42ec3e6cb070","Type":"ContainerStarted","Data":"b88a37e2305ada454ab96e32428d705135a942c4b78bf4e0c05fd91238115750"} Sep 08 11:30:13.675585 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.675561 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" event={"ID":"b03e1094-57b4-44d3-828f-f77b4a65a2b9","Type":"ContainerStarted","Data":"5c5c81f5739acf370752c1bf9d637da8aea4a968db853bf2be3c85edcca3f963"} Sep 08 11:30:13.679324 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.679299 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bjthr" event={"ID":"445df2c7-2335-4a07-bf1d-ad753e708cc3","Type":"ContainerStarted","Data":"c9ad916a2a774fa8ff4a84ecd5d3a45ff2393d81a9efba82a6e5ae5a88055b6f"} Sep 08 11:30:13.683131 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.683107 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zz9zq" event={"ID":"bec11709-37ed-4639-ae01-67ee4c64a5f7","Type":"ContainerStarted","Data":"5eaecc0725306d81ad9938625e90532c35d6aca4dad2c6927448cfdbd9a7bb0b"} Sep 08 11:30:13.686163 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.686100 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sb576" event={"ID":"c6a08843-d61b-451e-8e36-9e949a01059d","Type":"ContainerStarted","Data":"c9111336381d595441d4e092f2f1f1101a3e48094d4f531939bc46914897f159"} Sep 08 11:30:13.687855 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:13.687736 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" podUID="625b8b89-b76a-4075-901d-42ec3e6cb070" Sep 08 11:30:13.833245 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:13.833176 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 11:30:14.060995 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:14.060782 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 11:30:14.234362 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:14.234279 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:14.234536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:14.234420 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs\") pod \"network-metrics-daemon-rr4sq\" (UID: \"e3b04def-aa09-4e3b-ba69-8a8838695a83\") " pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:14.234609 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:14.234572 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 11:30:14.234664 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:14.234637 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs podName:e3b04def-aa09-4e3b-ba69-8a8838695a83 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:16.234618129 +0000 UTC m=+5.093805429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs") pod "network-metrics-daemon-rr4sq" (UID: "e3b04def-aa09-4e3b-ba69-8a8838695a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 11:30:14.235057 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:14.235032 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 11:30:14.235130 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:14.235094 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret podName:74934df5-b10d-497b-9ada-49aae488dd2b nodeName:}" failed. No retries permitted until 2026-09-08 11:30:16.235079491 +0000 UTC m=+5.094266788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret") pod "global-pull-secret-syncer-8r82d" (UID: "74934df5-b10d-497b-9ada-49aae488dd2b") : object "kube-system"/"original-pull-secret" not registered Sep 08 11:30:14.335675 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:14.335616 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbjj\" (UniqueName: \"kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj\") pod \"network-check-target-gbpbr\" (UID: \"16e669b2-9d2f-45ba-bb6a-447a8e87ae85\") " pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:14.335854 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:14.335842 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 11:30:14.335915 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:14.335861 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 11:30:14.335915 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:14.335874 2564 projected.go:196] Error preparing data for projected volume kube-api-access-hmbjj for pod openshift-network-diagnostics/network-check-target-gbpbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 11:30:14.336013 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:14.335930 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj podName:16e669b2-9d2f-45ba-bb6a-447a8e87ae85 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:16.335912886 +0000 UTC m=+5.195100197 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmbjj" (UniqueName: "kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj") pod "network-check-target-gbpbr" (UID: "16e669b2-9d2f-45ba-bb6a-447a8e87ae85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 11:30:14.566489 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:14.566400 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 11:25:12 +0000 UTC" deadline="2028-02-10 22:57:27.763663069 +0000 UTC" Sep 08 11:30:14.566489 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:14.566440 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12491h27m13.197227796s" Sep 08 11:30:14.647410 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:14.647364 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:14.647578 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:14.647518 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:14.647941 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:14.647922 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:14.648060 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:14.648024 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:15.647362 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:15.647328 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:15.647877 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:15.647479 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:16.253155 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:16.253119 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs\") pod \"network-metrics-daemon-rr4sq\" (UID: \"e3b04def-aa09-4e3b-ba69-8a8838695a83\") " pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:16.253155 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:16.253161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:16.253413 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:16.253266 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 11:30:16.253413 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:16.253322 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret podName:74934df5-b10d-497b-9ada-49aae488dd2b nodeName:}" failed. No retries permitted until 2026-09-08 11:30:20.253304703 +0000 UTC m=+9.112492022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret") pod "global-pull-secret-syncer-8r82d" (UID: "74934df5-b10d-497b-9ada-49aae488dd2b") : object "kube-system"/"original-pull-secret" not registered Sep 08 11:30:16.253413 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:16.253401 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 11:30:16.253592 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:16.253440 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs podName:e3b04def-aa09-4e3b-ba69-8a8838695a83 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:20.253427498 +0000 UTC m=+9.112614819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs") pod "network-metrics-daemon-rr4sq" (UID: "e3b04def-aa09-4e3b-ba69-8a8838695a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 11:30:16.354198 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:16.354102 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbjj\" (UniqueName: \"kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj\") pod \"network-check-target-gbpbr\" (UID: \"16e669b2-9d2f-45ba-bb6a-447a8e87ae85\") " pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:16.354346 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:16.354261 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 11:30:16.354346 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:16.354290 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 11:30:16.354346 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:16.354302 2564 projected.go:196] Error preparing data for projected volume kube-api-access-hmbjj for pod openshift-network-diagnostics/network-check-target-gbpbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 11:30:16.354533 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:16.354379 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj podName:16e669b2-9d2f-45ba-bb6a-447a8e87ae85 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:20.354345371 +0000 UTC m=+9.213532670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmbjj" (UniqueName: "kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj") pod "network-check-target-gbpbr" (UID: "16e669b2-9d2f-45ba-bb6a-447a8e87ae85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 11:30:16.647244 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:16.647157 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:16.647436 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:16.647296 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:16.647436 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:16.647419 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:16.647991 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:16.647527 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:17.647001 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:17.646959 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:17.647190 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:17.647115 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:18.646535 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:18.646500 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:18.647043 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:18.646635 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:18.647043 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:18.646683 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:18.647043 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:18.646814 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:19.647225 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:19.647194 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:19.647693 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:19.647336 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:20.282234 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:20.281992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs\") pod \"network-metrics-daemon-rr4sq\" (UID: \"e3b04def-aa09-4e3b-ba69-8a8838695a83\") " pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:20.282234 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:20.282047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:20.282234 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:20.282161 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 11:30:20.282234 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:20.282189 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 11:30:20.282234 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:20.282234 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs podName:e3b04def-aa09-4e3b-ba69-8a8838695a83 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:28.282215383 +0000 UTC m=+17.141402693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs") pod "network-metrics-daemon-rr4sq" (UID: "e3b04def-aa09-4e3b-ba69-8a8838695a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 11:30:20.282607 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:20.282251 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret podName:74934df5-b10d-497b-9ada-49aae488dd2b nodeName:}" failed. No retries permitted until 2026-09-08 11:30:28.282242996 +0000 UTC m=+17.141430296 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret") pod "global-pull-secret-syncer-8r82d" (UID: "74934df5-b10d-497b-9ada-49aae488dd2b") : object "kube-system"/"original-pull-secret" not registered Sep 08 11:30:20.383242 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:20.383203 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbjj\" (UniqueName: \"kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj\") pod \"network-check-target-gbpbr\" (UID: \"16e669b2-9d2f-45ba-bb6a-447a8e87ae85\") " pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:20.383944 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:20.383491 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 11:30:20.383944 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:20.383524 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 11:30:20.383944 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:20.383539 2564 projected.go:196] Error preparing data for projected volume kube-api-access-hmbjj for pod openshift-network-diagnostics/network-check-target-gbpbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 11:30:20.383944 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:20.383613 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj podName:16e669b2-9d2f-45ba-bb6a-447a8e87ae85 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:28.383592984 +0000 UTC m=+17.242780283 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmbjj" (UniqueName: "kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj") pod "network-check-target-gbpbr" (UID: "16e669b2-9d2f-45ba-bb6a-447a8e87ae85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 11:30:20.647527 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:20.647440 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:20.647527 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:20.647466 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:20.647990 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:20.647596 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:20.647990 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:20.647737 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:21.648027 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:21.647994 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:21.648492 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:21.648084 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:22.646888 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:22.646858 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:22.646888 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:22.646885 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:22.647118 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:22.646985 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:22.647173 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:22.647129 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:23.646773 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:23.646737 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:23.647195 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:23.646868 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:24.646526 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:24.646488 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:24.646526 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:24.646516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:24.647617 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:24.646783 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:24.647617 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:24.647531 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:25.647398 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:25.647357 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:25.647505 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:25.647468 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:26.649503 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:26.647628 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:26.649503 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:26.647647 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:26.649503 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:26.648013 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:26.649503 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:26.648070 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:26.725632 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:26.725589 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5dk5n" event={"ID":"42627895-7836-4cee-bb30-53fad710041e","Type":"ContainerStarted","Data":"eaba1fa8cdd9689980bbf7fba72b8b0cd8139adc55f7864504f4423ed8a14ba9"} Sep 08 11:30:26.729469 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:26.729043 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wfswx" event={"ID":"d96c6817-e9cc-47af-a268-2516ca461639","Type":"ContainerStarted","Data":"94e4232ea11d1269e62533ccaa63e2b2a46b4e6ea533b44c42d079d8ff4f330c"} Sep 08 11:30:26.731352 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:26.731324 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal" event={"ID":"18abe9a9a9441b764e0c8f71a4e8bdc5","Type":"ContainerStarted","Data":"063a144e34f5d288d8c90a1f23813bda30f7560bac401e8e02a73d7e28fbd771"} Sep 08 11:30:26.746713 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:26.746642 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5dk5n" podStartSLOduration=2.417479151 podStartE2EDuration="15.746620698s" totalImagesPullingTime="13.329141547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:26.746560728 +0000 UTC m=+15.605748046" watchObservedRunningTime="2026-09-08 11:30:26.746620698 +0000 UTC m=+15.605808017" Sep 08 11:30:26.768350 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:26.768269 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-187.ec2.internal" podStartSLOduration=14.768249637 podStartE2EDuration="14.768249637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:26.768097428 +0000 UTC m=+15.627284748" watchObservedRunningTime="2026-09-08 11:30:26.768249637 +0000 UTC m=+15.627436956" Sep 08 11:30:27.647160 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:27.647126 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:27.647338 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:27.647256 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:27.736428 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:27.736391 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zz9zq" event={"ID":"bec11709-37ed-4639-ae01-67ee4c64a5f7","Type":"ContainerStarted","Data":"5d7b4986d2751ca0137f906490e8517acf77a6a67fbedf1f7b58e48bd5939584"} Sep 08 11:30:27.738992 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:27.738964 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sb576" event={"ID":"c6a08843-d61b-451e-8e36-9e949a01059d","Type":"ContainerStarted","Data":"38147b5bf730389b637a7a1dc62df62ba8b0992059fb1a75a42697c6bb957049"} Sep 08 11:30:27.741199 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:27.740922 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6b84" event={"ID":"3eb89e07-e389-4c2f-9679-5867eecb4b2e","Type":"ContainerStarted","Data":"49a024c4013fa1a8ed476b065cec773ac1a0db37c8f03cd53352e5439dc60aff"} Sep 08 11:30:27.743171 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:27.743068 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5vslm" event={"ID":"15e73439-0f50-4c2e-9155-3e18ec9e5231","Type":"ContainerStarted","Data":"8bd2075d2c47afecfd230d6bf62e183f9643e23b4a8b1c90f59106d22cb4bd2f"} Sep 08 11:30:27.744847 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:27.744788 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" event={"ID":"b4035a6e430bc7a42c0e8baec502a110","Type":"ContainerStarted","Data":"e6105e192451d902718cb9ccc409d1b31de4445d1546714cba9c0771ab099644"} Sep 08 11:30:27.746402 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:27.746357 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" event={"ID":"b03e1094-57b4-44d3-828f-f77b4a65a2b9","Type":"ContainerStarted","Data":"d2d1a944b5d89cc9d422555a9da993c4cc256a9e84c444b96cce97cbebf14753"} Sep 08 11:30:27.748127 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:27.748103 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bjthr" event={"ID":"445df2c7-2335-4a07-bf1d-ad753e708cc3","Type":"ContainerStarted","Data":"4b831aa5dd5b13a6dfa5bc6a162c8583cd0654ff784626047bfe76f7b3fa30fb"} Sep 08 11:30:27.753213 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:27.753163 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wfswx" podStartSLOduration=3.939965869 podStartE2EDuration="16.753146105s" totalImagesPullingTime="12.813180236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:26.787856002 +0000 UTC m=+15.647043321" watchObservedRunningTime="2026-09-08 11:30:27.753146105 +0000 UTC m=+16.612333424" Sep 08 11:30:27.770249 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:27.770202 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sb576" podStartSLOduration=7.678305033 podStartE2EDuration="16.770185083s" totalImagesPullingTime="9.09188005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:27.769983255 +0000 UTC m=+16.629170601" watchObservedRunningTime="2026-09-08 11:30:27.770185083 +0000 UTC m=+16.629372407" Sep 08 11:30:27.770412 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:27.770385 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zz9zq" podStartSLOduration=3.9254868309999997 podStartE2EDuration="16.770359584s" totalImagesPullingTime="12.844872753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:27.752970921 +0000 UTC m=+16.612158239" watchObservedRunningTime="2026-09-08 11:30:27.770359584 +0000 UTC m=+16.629546905" Sep 08 11:30:27.871029 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:27.870965 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5vslm" podStartSLOduration=3.981857675 podStartE2EDuration="16.870947998s" totalImagesPullingTime="12.889090323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:27.852190968 +0000 UTC m=+16.711378310" watchObservedRunningTime="2026-09-08 11:30:27.870947998 +0000 UTC m=+16.730135315" Sep 08 11:30:28.340278 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:28.340187 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs\") pod \"network-metrics-daemon-rr4sq\" (UID: \"e3b04def-aa09-4e3b-ba69-8a8838695a83\") " pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:28.340278 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:28.340238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:28.340552 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:28.340378 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 11:30:28.340552 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:28.340393 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 11:30:28.340552 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:28.340445 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret podName:74934df5-b10d-497b-9ada-49aae488dd2b nodeName:}" failed. No retries permitted until 2026-09-08 11:30:44.340425897 +0000 UTC m=+33.199613208 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret") pod "global-pull-secret-syncer-8r82d" (UID: "74934df5-b10d-497b-9ada-49aae488dd2b") : object "kube-system"/"original-pull-secret" not registered Sep 08 11:30:28.340552 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:28.340465 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs podName:e3b04def-aa09-4e3b-ba69-8a8838695a83 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:44.340455213 +0000 UTC m=+33.199642508 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs") pod "network-metrics-daemon-rr4sq" (UID: "e3b04def-aa09-4e3b-ba69-8a8838695a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 11:30:28.441072 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:28.441040 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbjj\" (UniqueName: \"kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj\") pod \"network-check-target-gbpbr\" (UID: \"16e669b2-9d2f-45ba-bb6a-447a8e87ae85\") " pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:28.441264 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:28.441236 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 11:30:28.441322 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:28.441271 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 11:30:28.441322 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:28.441284 2564 projected.go:196] Error preparing data for projected volume kube-api-access-hmbjj for pod openshift-network-diagnostics/network-check-target-gbpbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 11:30:28.441425 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:28.441349 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj podName:16e669b2-9d2f-45ba-bb6a-447a8e87ae85 nodeName:}" failed. No retries permitted until 2026-09-08 11:30:44.441329129 +0000 UTC m=+33.300516428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmbjj" (UniqueName: "kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj") pod "network-check-target-gbpbr" (UID: "16e669b2-9d2f-45ba-bb6a-447a8e87ae85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 11:30:28.540065 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:28.540042 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 08 11:30:28.572021 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:28.571936 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T11:30:28.540061471Z","UUID":"d73a5117-9338-4886-befa-69318b1abc32","Handler":null,"Name":"","Endpoint":""} Sep 08 11:30:28.573709 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:28.573686 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 08 11:30:28.573822 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:28.573721 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 08 11:30:28.647252 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:28.647236 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:28.647393 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:28.647333 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:28.647449 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:28.647390 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:28.647491 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:28.647474 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:28.752118 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:28.752083 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" event={"ID":"b03e1094-57b4-44d3-828f-f77b4a65a2b9","Type":"ContainerStarted","Data":"5b3201a56d42a209f65e93bdc7d74d851d30ca85f371fbae8307aea710eb4c9a"} Sep 08 11:30:28.753717 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:28.753686 2564 generic.go:356] "Generic (PLEG): container finished" podID="3eb89e07-e389-4c2f-9679-5867eecb4b2e" containerID="49a024c4013fa1a8ed476b065cec773ac1a0db37c8f03cd53352e5439dc60aff" exitCode=0 Sep 08 11:30:28.753857 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:28.753771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6b84" event={"ID":"3eb89e07-e389-4c2f-9679-5867eecb4b2e","Type":"ContainerDied","Data":"49a024c4013fa1a8ed476b065cec773ac1a0db37c8f03cd53352e5439dc60aff"} Sep 08 11:30:28.755585 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:28.755562 2564 generic.go:356] "Generic (PLEG): container finished" podID="b4035a6e430bc7a42c0e8baec502a110" containerID="e6105e192451d902718cb9ccc409d1b31de4445d1546714cba9c0771ab099644" exitCode=0 Sep 08 11:30:28.755676 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:28.755645 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" event={"ID":"b4035a6e430bc7a42c0e8baec502a110","Type":"ContainerDied","Data":"e6105e192451d902718cb9ccc409d1b31de4445d1546714cba9c0771ab099644"} Sep 08 11:30:28.778855 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:28.778811 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bjthr" podStartSLOduration=4.977843391 podStartE2EDuration="17.778785522s" totalImagesPullingTime="12.800942131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:27.870463053 +0000 UTC m=+16.729650389" watchObservedRunningTime="2026-09-08 11:30:28.778785522 +0000 UTC m=+17.637972842" Sep 08 11:30:29.647359 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:29.647126 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:29.647545 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:29.647470 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:29.760720 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:29.760623 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" event={"ID":"b03e1094-57b4-44d3-828f-f77b4a65a2b9","Type":"ContainerStarted","Data":"094237bf9805bbd001101417b94edd619958c528233a38253c2cb41679053c4c"} Sep 08 11:30:29.762990 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:29.762963 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" event={"ID":"b4035a6e430bc7a42c0e8baec502a110","Type":"ContainerStarted","Data":"5b979eb3088de9d26c8d26e9927d0f390d12dfc1766411955fe8433fe9978c95"} Sep 08 11:30:29.799485 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:29.799430 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-187.ec2.internal" podStartSLOduration=17.799410739 podStartE2EDuration="17.799410739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:29.799039128 +0000 UTC m=+18.658226447" watchObservedRunningTime="2026-09-08 11:30:29.799410739 +0000 UTC m=+18.658598060" Sep 08 11:30:29.799814 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:29.799774 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pwzxb" podStartSLOduration=3.80749533 podStartE2EDuration="18.79976454s" totalImagesPullingTime="14.99226921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:30:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:29.782717641 +0000 UTC m=+18.641904960" watchObservedRunningTime="2026-09-08 11:30:29.79976454 +0000 UTC m=+18.658951858" Sep 08 11:30:30.646832 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:30.646801 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:30.647018 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:30.646938 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:30.647018 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:30.646955 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:30.647123 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:30.647071 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:31.647895 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:31.647863 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:31.648643 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:31.647976 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:31.733471 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:31.733437 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bjthr" Sep 08 11:30:31.734056 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:31.734033 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bjthr" Sep 08 11:30:32.647294 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:32.647262 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:32.647294 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:32.647281 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:32.647567 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:32.647412 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:32.647567 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:32.647529 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:33.647409 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:33.647379 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:33.647851 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:33.647481 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:34.646796 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:34.646757 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:34.646985 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:34.646769 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:34.646985 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:34.646910 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:34.647106 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:34.646989 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:35.646953 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:35.646918 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:35.647434 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:35.647069 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:36.646916 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:36.646887 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:36.647089 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:36.646988 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:36.647089 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:36.647070 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:36.647409 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:36.647154 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:37.647310 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:37.647085 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:37.647779 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:37.647432 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:37.782400 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:37.782352 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" event={"ID":"625b8b89-b76a-4075-901d-42ec3e6cb070","Type":"ContainerStarted","Data":"6852999cbe44f0257834cc24929a6f19960aaf1226fd8eaebd80d8535b0333f1"} Sep 08 11:30:37.782533 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:37.782452 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" event={"ID":"625b8b89-b76a-4075-901d-42ec3e6cb070","Type":"ContainerStarted","Data":"1f5ff27ae94f26f900e9faabe208b3c439bb7090177ba8e78c737a7b5a3d6e0f"} Sep 08 11:30:37.782533 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:37.782489 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" event={"ID":"625b8b89-b76a-4075-901d-42ec3e6cb070","Type":"ContainerStarted","Data":"7c100f62ef163866346ea8952bbe73dd84fbdc7ab3722d9b652bc693b9884865"} Sep 08 11:30:37.782533 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:37.782520 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" event={"ID":"625b8b89-b76a-4075-901d-42ec3e6cb070","Type":"ContainerStarted","Data":"fcb2a8727a63ad30eec80cd09ee698ee9263c51ef8434128772c0150fec7cda3"} Sep 08 11:30:37.782629 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:37.782550 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" event={"ID":"625b8b89-b76a-4075-901d-42ec3e6cb070","Type":"ContainerStarted","Data":"4f3f36e3541f73e7133d0d9710a849b6120c14e0bc664fafa20d46cdc2cf1a5c"} Sep 08 11:30:37.782629 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:37.782596 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" event={"ID":"625b8b89-b76a-4075-901d-42ec3e6cb070","Type":"ContainerStarted","Data":"61c0396dba98c9313923e1d1cc49c4912960c6e8eddcf9830791300197d6fda0"} Sep 08 11:30:37.784224 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:37.784203 2564 generic.go:356] "Generic (PLEG): container finished" podID="3eb89e07-e389-4c2f-9679-5867eecb4b2e" containerID="9186ce10aee70507fb61d2fa8f8fc317295e532662c5e6b12569faf062b73d93" exitCode=0 Sep 08 11:30:37.784286 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:37.784242 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6b84" event={"ID":"3eb89e07-e389-4c2f-9679-5867eecb4b2e","Type":"ContainerDied","Data":"9186ce10aee70507fb61d2fa8f8fc317295e532662c5e6b12569faf062b73d93"} Sep 08 11:30:38.647145 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:38.646975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:38.647312 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:38.647024 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:38.647312 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:38.647218 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:38.647629 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:38.647314 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:39.647432 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:39.647412 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:39.647773 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:39.647513 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:39.791834 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:39.791791 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" event={"ID":"625b8b89-b76a-4075-901d-42ec3e6cb070","Type":"ContainerStarted","Data":"ce30bef241eeae37d4a745b1845751114ed1caf756d2abe1faf610bd70151e0c"} Sep 08 11:30:39.793940 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:39.793916 2564 generic.go:356] "Generic (PLEG): container finished" podID="3eb89e07-e389-4c2f-9679-5867eecb4b2e" containerID="64506f82eae1cd6f68e80bbc2dc1b513018f83f003b921b1845961315636ea47" exitCode=0 Sep 08 11:30:39.794040 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:39.793951 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6b84" event={"ID":"3eb89e07-e389-4c2f-9679-5867eecb4b2e","Type":"ContainerDied","Data":"64506f82eae1cd6f68e80bbc2dc1b513018f83f003b921b1845961315636ea47"} Sep 08 11:30:40.646755 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:40.646724 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:40.646870 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:40.646821 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:40.646912 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:40.646877 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:40.646976 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:40.646960 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:40.798547 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:40.798478 2564 generic.go:356] "Generic (PLEG): container finished" podID="3eb89e07-e389-4c2f-9679-5867eecb4b2e" containerID="7564f446372c8bf0ce2ea161d5078b52b0624a005b29d310abacc3e4dc09f0f6" exitCode=0 Sep 08 11:30:40.798547 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:40.798521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6b84" event={"ID":"3eb89e07-e389-4c2f-9679-5867eecb4b2e","Type":"ContainerDied","Data":"7564f446372c8bf0ce2ea161d5078b52b0624a005b29d310abacc3e4dc09f0f6"} Sep 08 11:30:41.648400 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:41.648351 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:41.648565 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:41.648481 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:41.805033 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:41.804996 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" event={"ID":"625b8b89-b76a-4075-901d-42ec3e6cb070","Type":"ContainerStarted","Data":"2197eff39cb77b294e77b7697132cb29a90b64d583d56ef221e2f0a19e4abd6b"} Sep 08 11:30:41.805771 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:41.805272 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:41.805771 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:41.805332 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:41.805771 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:41.805394 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:41.821597 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:41.821575 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:41.821732 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:41.821693 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:30:41.888739 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:41.888687 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" podStartSLOduration=7.142737954 podStartE2EDuration="30.888673432s" totalImagesPullingTime="23.745935478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=2 observedRunningTime="2026-09-08 11:30:41.846519889 +0000 UTC m=+30.705707226" watchObservedRunningTime="2026-09-08 11:30:41.888673432 +0000 UTC m=+30.747860749" Sep 08 11:30:41.914425 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:41.914333 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bjthr" Sep 08 11:30:41.914562 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:41.914484 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 08 11:30:41.914875 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:41.914856 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bjthr" Sep 08 11:30:42.647789 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:42.647025 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:42.647789 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:42.647184 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:42.647789 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:42.647630 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:42.647789 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:42.647723 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:43.568190 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:43.568157 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gbpbr"] Sep 08 11:30:43.569136 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:43.568292 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:43.569136 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:43.568420 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:43.571506 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:43.571478 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8r82d"] Sep 08 11:30:43.571706 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:43.571561 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:43.571706 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:43.571647 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:43.572106 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:43.572084 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rr4sq"] Sep 08 11:30:43.572203 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:43.572173 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:43.572304 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:43.572281 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:44.354572 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:44.354534 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs\") pod \"network-metrics-daemon-rr4sq\" (UID: \"e3b04def-aa09-4e3b-ba69-8a8838695a83\") " pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:44.354849 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:44.354582 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:44.354849 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:44.354699 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 11:30:44.354849 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:44.354773 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret podName:74934df5-b10d-497b-9ada-49aae488dd2b nodeName:}" failed. No retries permitted until 2026-09-08 11:31:16.354753366 +0000 UTC m=+65.213940712 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret") pod "global-pull-secret-syncer-8r82d" (UID: "74934df5-b10d-497b-9ada-49aae488dd2b") : object "kube-system"/"original-pull-secret" not registered Sep 08 11:30:44.354849 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:44.354700 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 11:30:44.355050 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:44.354878 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs podName:e3b04def-aa09-4e3b-ba69-8a8838695a83 nodeName:}" failed. No retries permitted until 2026-09-08 11:31:16.354858941 +0000 UTC m=+65.214046249 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs") pod "network-metrics-daemon-rr4sq" (UID: "e3b04def-aa09-4e3b-ba69-8a8838695a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 11:30:44.455665 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:44.455627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbjj\" (UniqueName: \"kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj\") pod \"network-check-target-gbpbr\" (UID: \"16e669b2-9d2f-45ba-bb6a-447a8e87ae85\") " pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:44.455858 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:44.455760 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 11:30:44.455858 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:44.455788 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 11:30:44.455858 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:44.455801 2564 projected.go:196] Error preparing data for projected volume kube-api-access-hmbjj for pod openshift-network-diagnostics/network-check-target-gbpbr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 11:30:44.456010 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:44.455860 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj podName:16e669b2-9d2f-45ba-bb6a-447a8e87ae85 nodeName:}" failed. No retries permitted until 2026-09-08 11:31:16.455842284 +0000 UTC m=+65.315029594 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmbjj" (UniqueName: "kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj") pod "network-check-target-gbpbr" (UID: "16e669b2-9d2f-45ba-bb6a-447a8e87ae85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 11:30:45.647633 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:45.647600 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:45.647633 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:45.647627 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:45.648147 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:45.647600 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:45.648147 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:45.647732 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:45.648147 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:45.647802 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:45.648147 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:45.647888 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:46.818714 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:46.818650 2564 generic.go:356] "Generic (PLEG): container finished" podID="3eb89e07-e389-4c2f-9679-5867eecb4b2e" containerID="a6ac84f8fa168df3a6b340e1f31417f931a7d3fadff1e1575e0a302032bd50f0" exitCode=0 Sep 08 11:30:46.819014 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:46.818712 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6b84" event={"ID":"3eb89e07-e389-4c2f-9679-5867eecb4b2e","Type":"ContainerDied","Data":"a6ac84f8fa168df3a6b340e1f31417f931a7d3fadff1e1575e0a302032bd50f0"} Sep 08 11:30:47.647519 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:47.647486 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:47.647519 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:47.647520 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:47.647703 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:47.647487 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:47.647703 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:47.647599 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r82d" podUID="74934df5-b10d-497b-9ada-49aae488dd2b" Sep 08 11:30:47.647770 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:47.647712 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbpbr" podUID="16e669b2-9d2f-45ba-bb6a-447a8e87ae85" Sep 08 11:30:47.647828 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:47.647812 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rr4sq" podUID="e3b04def-aa09-4e3b-ba69-8a8838695a83" Sep 08 11:30:47.825436 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:47.825345 2564 generic.go:356] "Generic (PLEG): container finished" podID="3eb89e07-e389-4c2f-9679-5867eecb4b2e" containerID="4c150a21500e9733601779ef2ccb1731f05e6c2371f557c0345832df39435757" exitCode=0 Sep 08 11:30:47.825436 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:47.825401 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6b84" event={"ID":"3eb89e07-e389-4c2f-9679-5867eecb4b2e","Type":"ContainerDied","Data":"4c150a21500e9733601779ef2ccb1731f05e6c2371f557c0345832df39435757"} Sep 08 11:30:48.830749 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:48.830579 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m6b84" event={"ID":"3eb89e07-e389-4c2f-9679-5867eecb4b2e","Type":"ContainerStarted","Data":"e6873e19de37696984cf3b726def5b1ddc8551d052d6eb00c6653e9d4506da5f"} Sep 08 11:30:48.859602 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:48.859564 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m6b84" podStartSLOduration=9.76505105 podStartE2EDuration="37.859550424s" totalImagesPullingTime="28.094499374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:30:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:48.859234063 +0000 UTC m=+37.718421400" watchObservedRunningTime="2026-09-08 11:30:48.859550424 +0000 UTC m=+37.718737741" Sep 08 11:30:49.370426 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.370401 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-187.ec2.internal" event="NodeReady" Sep 08 11:30:49.370582 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.370534 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 08 11:30:49.420865 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.420837 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-66f5s"] Sep 08 11:30:49.451306 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.451281 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lfp5b"] Sep 08 11:30:49.451463 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.451411 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:49.454110 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.454086 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 08 11:30:49.454519 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.454502 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2v99l\"" Sep 08 11:30:49.454607 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.454597 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 08 11:30:49.475326 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.475304 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-66f5s"] Sep 08 11:30:49.475431 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.475336 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lfp5b"] Sep 08 11:30:49.475475 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.475427 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lfp5b" Sep 08 11:30:49.478202 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.478181 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 08 11:30:49.478202 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.478202 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 08 11:30:49.478479 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.478467 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h4mlz\"" Sep 08 11:30:49.478545 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.478496 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 08 11:30:49.592393 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.592336 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/645f6f42-ec34-4b66-bf8e-183601ddc1af-config-volume\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:49.592563 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.592401 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7tmz\" (UniqueName: \"kubernetes.io/projected/645f6f42-ec34-4b66-bf8e-183601ddc1af-kube-api-access-b7tmz\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:49.592563 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.592447 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645f6f42-ec34-4b66-bf8e-183601ddc1af-tmp-dir\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:49.592563 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.592474 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwgp4\" (UniqueName: \"kubernetes.io/projected/535465a6-a1fc-4d49-917c-6bff81183ae9-kube-api-access-hwgp4\") pod \"ingress-canary-lfp5b\" (UID: \"535465a6-a1fc-4d49-917c-6bff81183ae9\") " pod="openshift-ingress-canary/ingress-canary-lfp5b" Sep 08 11:30:49.592563 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.592519 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/535465a6-a1fc-4d49-917c-6bff81183ae9-cert\") pod \"ingress-canary-lfp5b\" (UID: \"535465a6-a1fc-4d49-917c-6bff81183ae9\") " pod="openshift-ingress-canary/ingress-canary-lfp5b" Sep 08 11:30:49.592694 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.592585 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:49.646615 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.646587 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:30:49.646770 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.646588 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:30:49.646770 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.646698 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:30:49.650249 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.650226 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 11:30:49.650491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.650473 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 11:30:49.650592 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.650489 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xt4h2\"" Sep 08 11:30:49.650788 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.650771 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 11:30:49.650875 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.650809 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nwz6n\"" Sep 08 11:30:49.650875 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.650852 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 11:30:49.693060 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.693035 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7tmz\" (UniqueName: \"kubernetes.io/projected/645f6f42-ec34-4b66-bf8e-183601ddc1af-kube-api-access-b7tmz\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:49.693135 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.693078 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645f6f42-ec34-4b66-bf8e-183601ddc1af-tmp-dir\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:49.693135 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.693099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwgp4\" (UniqueName: \"kubernetes.io/projected/535465a6-a1fc-4d49-917c-6bff81183ae9-kube-api-access-hwgp4\") pod \"ingress-canary-lfp5b\" (UID: \"535465a6-a1fc-4d49-917c-6bff81183ae9\") " pod="openshift-ingress-canary/ingress-canary-lfp5b" Sep 08 11:30:49.693135 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.693124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/535465a6-a1fc-4d49-917c-6bff81183ae9-cert\") pod \"ingress-canary-lfp5b\" (UID: \"535465a6-a1fc-4d49-917c-6bff81183ae9\") " pod="openshift-ingress-canary/ingress-canary-lfp5b" Sep 08 11:30:49.693296 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.693180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:49.693296 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:49.693289 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 11:30:49.693413 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:49.693345 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls podName:645f6f42-ec34-4b66-bf8e-183601ddc1af nodeName:}" failed. No retries permitted until 2026-09-08 11:30:50.193327354 +0000 UTC m=+39.052514653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls") pod "dns-default-66f5s" (UID: "645f6f42-ec34-4b66-bf8e-183601ddc1af") : secret "dns-default-metrics-tls" not found Sep 08 11:30:49.693473 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.693422 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645f6f42-ec34-4b66-bf8e-183601ddc1af-tmp-dir\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:49.693536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.693478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/645f6f42-ec34-4b66-bf8e-183601ddc1af-config-volume\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:49.693942 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.693924 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/645f6f42-ec34-4b66-bf8e-183601ddc1af-config-volume\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:49.696923 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.696903 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/535465a6-a1fc-4d49-917c-6bff81183ae9-cert\") pod \"ingress-canary-lfp5b\" (UID: \"535465a6-a1fc-4d49-917c-6bff81183ae9\") " pod="openshift-ingress-canary/ingress-canary-lfp5b" Sep 08 11:30:49.702247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.702222 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwgp4\" (UniqueName: \"kubernetes.io/projected/535465a6-a1fc-4d49-917c-6bff81183ae9-kube-api-access-hwgp4\") pod \"ingress-canary-lfp5b\" (UID: \"535465a6-a1fc-4d49-917c-6bff81183ae9\") " pod="openshift-ingress-canary/ingress-canary-lfp5b" Sep 08 11:30:49.702320 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.702304 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7tmz\" (UniqueName: \"kubernetes.io/projected/645f6f42-ec34-4b66-bf8e-183601ddc1af-kube-api-access-b7tmz\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:49.783464 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:49.783436 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lfp5b" Sep 08 11:30:50.104829 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:50.104759 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lfp5b"] Sep 08 11:30:50.108886 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:30:50.108861 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535465a6_a1fc_4d49_917c_6bff81183ae9.slice/crio-0433e819e3b68f05e463f2836693f4df8a79b7ead97726539cc053d5629d5804 WatchSource:0}: Error finding container 0433e819e3b68f05e463f2836693f4df8a79b7ead97726539cc053d5629d5804: Status 404 returned error can't find the container with id 0433e819e3b68f05e463f2836693f4df8a79b7ead97726539cc053d5629d5804 Sep 08 11:30:50.198147 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:50.198113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:50.198287 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:50.198268 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 11:30:50.198342 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:50.198331 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls podName:645f6f42-ec34-4b66-bf8e-183601ddc1af nodeName:}" failed. No retries permitted until 2026-09-08 11:30:51.198315075 +0000 UTC m=+40.057502372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls") pod "dns-default-66f5s" (UID: "645f6f42-ec34-4b66-bf8e-183601ddc1af") : secret "dns-default-metrics-tls" not found Sep 08 11:30:50.837287 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:50.837244 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lfp5b" event={"ID":"535465a6-a1fc-4d49-917c-6bff81183ae9","Type":"ContainerStarted","Data":"0433e819e3b68f05e463f2836693f4df8a79b7ead97726539cc053d5629d5804"} Sep 08 11:30:51.204838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:51.204805 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:51.205284 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:51.204983 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 11:30:51.205284 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:51.205051 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls podName:645f6f42-ec34-4b66-bf8e-183601ddc1af nodeName:}" failed. No retries permitted until 2026-09-08 11:30:53.205035085 +0000 UTC m=+42.064222400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls") pod "dns-default-66f5s" (UID: "645f6f42-ec34-4b66-bf8e-183601ddc1af") : secret "dns-default-metrics-tls" not found Sep 08 11:30:53.221559 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:53.221517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:53.222000 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:53.221717 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 11:30:53.222000 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:53.221804 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls podName:645f6f42-ec34-4b66-bf8e-183601ddc1af nodeName:}" failed. No retries permitted until 2026-09-08 11:30:57.221780102 +0000 UTC m=+46.080967415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls") pod "dns-default-66f5s" (UID: "645f6f42-ec34-4b66-bf8e-183601ddc1af") : secret "dns-default-metrics-tls" not found Sep 08 11:30:53.845887 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:53.845810 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lfp5b" event={"ID":"535465a6-a1fc-4d49-917c-6bff81183ae9","Type":"ContainerStarted","Data":"dd23fccf9c0ff9b46c51edcd8a2cfb4b0244015075b5258ea62df0b07d5f460a"} Sep 08 11:30:53.864029 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:53.863983 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lfp5b" podStartSLOduration=1.990685195 podStartE2EDuration="4.863969398s" totalImagesPullingTime="2.873284203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:30:53.863540314 +0000 UTC m=+42.722727645" watchObservedRunningTime="2026-09-08 11:30:53.863969398 +0000 UTC m=+42.723156716" Sep 08 11:30:57.246484 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:30:57.246435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:30:57.246884 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:57.246589 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 11:30:57.246884 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:30:57.246655 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls podName:645f6f42-ec34-4b66-bf8e-183601ddc1af nodeName:}" failed. No retries permitted until 2026-09-08 11:31:05.246638952 +0000 UTC m=+54.105826247 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls") pod "dns-default-66f5s" (UID: "645f6f42-ec34-4b66-bf8e-183601ddc1af") : secret "dns-default-metrics-tls" not found Sep 08 11:31:05.299603 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:05.299552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:31:05.300031 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:05.299704 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 11:31:05.300031 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:05.299767 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls podName:645f6f42-ec34-4b66-bf8e-183601ddc1af nodeName:}" failed. No retries permitted until 2026-09-08 11:31:21.299751929 +0000 UTC m=+70.158939224 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls") pod "dns-default-66f5s" (UID: "645f6f42-ec34-4b66-bf8e-183601ddc1af") : secret "dns-default-metrics-tls" not found Sep 08 11:31:13.819957 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:13.819926 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5x6mb" Sep 08 11:31:16.371840 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:16.371802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs\") pod \"network-metrics-daemon-rr4sq\" (UID: \"e3b04def-aa09-4e3b-ba69-8a8838695a83\") " pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:31:16.371840 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:16.371845 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:31:16.374815 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:16.374790 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 11:31:16.374926 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:16.374823 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 11:31:16.382450 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:16.382429 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 08 11:31:16.382513 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:16.382500 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs podName:e3b04def-aa09-4e3b-ba69-8a8838695a83 nodeName:}" failed. No retries permitted until 2026-09-08 11:32:20.382478739 +0000 UTC m=+129.241666051 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs") pod "network-metrics-daemon-rr4sq" (UID: "e3b04def-aa09-4e3b-ba69-8a8838695a83") : secret "metrics-daemon-secret" not found Sep 08 11:31:16.385351 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:16.385332 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74934df5-b10d-497b-9ada-49aae488dd2b-original-pull-secret\") pod \"global-pull-secret-syncer-8r82d\" (UID: \"74934df5-b10d-497b-9ada-49aae488dd2b\") " pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:31:16.472814 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:16.472786 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbjj\" (UniqueName: \"kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj\") pod \"network-check-target-gbpbr\" (UID: \"16e669b2-9d2f-45ba-bb6a-447a8e87ae85\") " pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:31:16.475926 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:16.475908 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 11:31:16.486398 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:16.486382 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 11:31:16.496997 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:16.496978 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmbjj\" (UniqueName: \"kubernetes.io/projected/16e669b2-9d2f-45ba-bb6a-447a8e87ae85-kube-api-access-hmbjj\") pod \"network-check-target-gbpbr\" (UID: \"16e669b2-9d2f-45ba-bb6a-447a8e87ae85\") " pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:31:16.657128 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:16.657106 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r82d" Sep 08 11:31:16.665820 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:16.665799 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nwz6n\"" Sep 08 11:31:16.673827 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:16.673812 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:31:16.972064 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:16.972005 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8r82d"] Sep 08 11:31:16.974062 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:31:16.974029 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74934df5_b10d_497b_9ada_49aae488dd2b.slice/crio-e0799c53d3a99cd7659b7030fa6175bf23ff8c748dbc65da0d635d598fb8bd23 WatchSource:0}: Error finding container e0799c53d3a99cd7659b7030fa6175bf23ff8c748dbc65da0d635d598fb8bd23: Status 404 returned error can't find the container with id e0799c53d3a99cd7659b7030fa6175bf23ff8c748dbc65da0d635d598fb8bd23 Sep 08 11:31:16.990712 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:16.990689 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gbpbr"] Sep 08 11:31:16.993393 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:31:16.993356 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e669b2_9d2f_45ba_bb6a_447a8e87ae85.slice/crio-dfc8dc46b64c4b4bba74cd066cf6537b429c87c622d5287679618d165e41a10b WatchSource:0}: Error finding container dfc8dc46b64c4b4bba74cd066cf6537b429c87c622d5287679618d165e41a10b: Status 404 returned error can't find the container with id dfc8dc46b64c4b4bba74cd066cf6537b429c87c622d5287679618d165e41a10b Sep 08 11:31:17.907134 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:17.906945 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8r82d" event={"ID":"74934df5-b10d-497b-9ada-49aae488dd2b","Type":"ContainerStarted","Data":"e0799c53d3a99cd7659b7030fa6175bf23ff8c748dbc65da0d635d598fb8bd23"} Sep 08 11:31:17.908687 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:17.908654 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gbpbr" event={"ID":"16e669b2-9d2f-45ba-bb6a-447a8e87ae85","Type":"ContainerStarted","Data":"dfc8dc46b64c4b4bba74cd066cf6537b429c87c622d5287679618d165e41a10b"} Sep 08 11:31:21.306928 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:21.306884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:31:21.307383 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:21.307062 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 11:31:21.307383 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:21.307158 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls podName:645f6f42-ec34-4b66-bf8e-183601ddc1af nodeName:}" failed. No retries permitted until 2026-09-08 11:31:53.307135818 +0000 UTC m=+102.166323114 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls") pod "dns-default-66f5s" (UID: "645f6f42-ec34-4b66-bf8e-183601ddc1af") : secret "dns-default-metrics-tls" not found Sep 08 11:31:21.921300 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:21.921259 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gbpbr" event={"ID":"16e669b2-9d2f-45ba-bb6a-447a8e87ae85","Type":"ContainerStarted","Data":"b52ca960029ca9f032eb2af7397ec5d64e4665f8964e3c77dcde065d7a06cc81"} Sep 08 11:31:21.921489 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:21.921384 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:31:21.922777 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:21.922743 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8r82d" event={"ID":"74934df5-b10d-497b-9ada-49aae488dd2b","Type":"ContainerStarted","Data":"4d8f285afce8d2e8a1a3b53ae5fcb9dab06fd51db5e16d39d3b6954fd08f6c7a"} Sep 08 11:31:21.939795 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:21.939751 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gbpbr" podStartSLOduration=66.490583368 podStartE2EDuration="1m10.939734773s" totalImagesPullingTime="4.449151405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:31:21.939339485 +0000 UTC m=+70.798526823" watchObservedRunningTime="2026-09-08 11:31:21.939734773 +0000 UTC m=+70.798922086" Sep 08 11:31:21.957716 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:21.957680 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8r82d" podStartSLOduration=66.486029501 podStartE2EDuration="1m10.95766777s" totalImagesPullingTime="4.471638269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:31:21.957438113 +0000 UTC m=+70.816625441" watchObservedRunningTime="2026-09-08 11:31:21.95766777 +0000 UTC m=+70.816855088" Sep 08 11:31:43.489350 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.489241 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5"] Sep 08 11:31:43.493943 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.493916 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" Sep 08 11:31:43.496551 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.496524 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 08 11:31:43.496551 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.496540 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 08 11:31:43.496726 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.496538 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 08 11:31:43.497705 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.497688 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-b7kq4\"" Sep 08 11:31:43.497804 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.497766 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 08 11:31:43.501485 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.501465 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5"] Sep 08 11:31:43.597426 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.597389 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-p5jkx"] Sep 08 11:31:43.600812 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.600787 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-2mxp7"] Sep 08 11:31:43.600952 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.600904 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.603880 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.603862 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 08 11:31:43.603996 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.603882 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 08 11:31:43.603996 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.603929 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rdtt7\"" Sep 08 11:31:43.604198 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.604182 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-545d764448-lqfvt"] Sep 08 11:31:43.604310 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.604296 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:43.604431 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.604411 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 08 11:31:43.604503 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.604485 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 08 11:31:43.606805 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.606788 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.607233 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.607213 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 08 11:31:43.607982 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.607963 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 08 11:31:43.608112 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.608046 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9nbx8\"" Sep 08 11:31:43.608112 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.608101 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 08 11:31:43.608536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.608519 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 08 11:31:43.610696 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.610677 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 08 11:31:43.610774 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.610720 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 08 11:31:43.611755 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.611739 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-59gmm\"" Sep 08 11:31:43.611755 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.611754 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 08 11:31:43.612960 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.612943 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 08 11:31:43.617987 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.617966 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-p5jkx"] Sep 08 11:31:43.618358 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.618338 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 08 11:31:43.618808 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.618792 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 08 11:31:43.619617 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.619597 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-545d764448-lqfvt"] Sep 08 11:31:43.630902 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.630882 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-2mxp7"] Sep 08 11:31:43.650786 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.650761 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv2g6\" (UniqueName: \"kubernetes.io/projected/de336abd-d8f8-429f-ab55-0041367141c5-kube-api-access-mv2g6\") pod \"service-ca-operator-859f94c4b-zspv5\" (UID: \"de336abd-d8f8-429f-ab55-0041367141c5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" Sep 08 11:31:43.650877 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.650801 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de336abd-d8f8-429f-ab55-0041367141c5-serving-cert\") pod \"service-ca-operator-859f94c4b-zspv5\" (UID: \"de336abd-d8f8-429f-ab55-0041367141c5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" Sep 08 11:31:43.650877 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.650822 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de336abd-d8f8-429f-ab55-0041367141c5-config\") pod \"service-ca-operator-859f94c4b-zspv5\" (UID: \"de336abd-d8f8-429f-ab55-0041367141c5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" Sep 08 11:31:43.751173 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/660f1576-9451-4015-b19c-640193d50ccc-image-registry-private-configuration\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.751173 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv2g6\" (UniqueName: \"kubernetes.io/projected/de336abd-d8f8-429f-ab55-0041367141c5-kube-api-access-mv2g6\") pod \"service-ca-operator-859f94c4b-zspv5\" (UID: \"de336abd-d8f8-429f-ab55-0041367141c5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" Sep 08 11:31:43.751173 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751151 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-bound-sa-token\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.751173 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44bd9c6-f403-43f1-820f-e49f9ded15e6-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.751491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f44bd9c6-f403-43f1-820f-e49f9ded15e6-snapshots\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.751491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751201 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4g7t\" (UniqueName: \"kubernetes.io/projected/f44bd9c6-f403-43f1-820f-e49f9ded15e6-kube-api-access-z4g7t\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.751491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751217 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xvc4\" (UniqueName: \"kubernetes.io/projected/a3af17fe-667d-4032-9a0e-aaadfcdacb2f-kube-api-access-7xvc4\") pod \"console-operator-85b454874b-2mxp7\" (UID: \"a3af17fe-667d-4032-9a0e-aaadfcdacb2f\") " pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:43.751491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/660f1576-9451-4015-b19c-640193d50ccc-trusted-ca\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.751491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44bd9c6-f403-43f1-820f-e49f9ded15e6-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.751491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751270 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3af17fe-667d-4032-9a0e-aaadfcdacb2f-serving-cert\") pod \"console-operator-85b454874b-2mxp7\" (UID: \"a3af17fe-667d-4032-9a0e-aaadfcdacb2f\") " pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:43.751491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751291 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de336abd-d8f8-429f-ab55-0041367141c5-serving-cert\") pod \"service-ca-operator-859f94c4b-zspv5\" (UID: \"de336abd-d8f8-429f-ab55-0041367141c5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" Sep 08 11:31:43.751491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5nz2\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-kube-api-access-f5nz2\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.751491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751320 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f44bd9c6-f403-43f1-820f-e49f9ded15e6-tmp\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.751491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751335 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3af17fe-667d-4032-9a0e-aaadfcdacb2f-trusted-ca\") pod \"console-operator-85b454874b-2mxp7\" (UID: \"a3af17fe-667d-4032-9a0e-aaadfcdacb2f\") " pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:43.751491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751393 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/660f1576-9451-4015-b19c-640193d50ccc-ca-trust-extracted\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.751491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751413 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3af17fe-667d-4032-9a0e-aaadfcdacb2f-config\") pod \"console-operator-85b454874b-2mxp7\" (UID: \"a3af17fe-667d-4032-9a0e-aaadfcdacb2f\") " pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:43.751491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/660f1576-9451-4015-b19c-640193d50ccc-installation-pull-secrets\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.751491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de336abd-d8f8-429f-ab55-0041367141c5-config\") pod \"service-ca-operator-859f94c4b-zspv5\" (UID: \"de336abd-d8f8-429f-ab55-0041367141c5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" Sep 08 11:31:43.751965 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751498 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f44bd9c6-f403-43f1-820f-e49f9ded15e6-serving-cert\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.751965 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/660f1576-9451-4015-b19c-640193d50ccc-registry-certificates\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.751965 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.751752 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.752057 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.752015 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de336abd-d8f8-429f-ab55-0041367141c5-config\") pod \"service-ca-operator-859f94c4b-zspv5\" (UID: \"de336abd-d8f8-429f-ab55-0041367141c5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" Sep 08 11:31:43.753486 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.753467 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de336abd-d8f8-429f-ab55-0041367141c5-serving-cert\") pod \"service-ca-operator-859f94c4b-zspv5\" (UID: \"de336abd-d8f8-429f-ab55-0041367141c5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" Sep 08 11:31:43.759337 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.759316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv2g6\" (UniqueName: \"kubernetes.io/projected/de336abd-d8f8-429f-ab55-0041367141c5-kube-api-access-mv2g6\") pod \"service-ca-operator-859f94c4b-zspv5\" (UID: \"de336abd-d8f8-429f-ab55-0041367141c5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" Sep 08 11:31:43.803328 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.803306 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" Sep 08 11:31:43.852491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.852463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.852579 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.852526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/660f1576-9451-4015-b19c-640193d50ccc-image-registry-private-configuration\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.852579 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.852564 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-bound-sa-token\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.852682 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.852587 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44bd9c6-f403-43f1-820f-e49f9ded15e6-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.852682 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.852613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f44bd9c6-f403-43f1-820f-e49f9ded15e6-snapshots\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.852682 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.852638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4g7t\" (UniqueName: \"kubernetes.io/projected/f44bd9c6-f403-43f1-820f-e49f9ded15e6-kube-api-access-z4g7t\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.852682 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:43.852653 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 11:31:43.852682 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:43.852678 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-545d764448-lqfvt: secret "image-registry-tls" not found Sep 08 11:31:43.852926 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:43.852754 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls podName:660f1576-9451-4015-b19c-640193d50ccc nodeName:}" failed. No retries permitted until 2026-09-08 11:31:44.352732376 +0000 UTC m=+93.211919695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls") pod "image-registry-545d764448-lqfvt" (UID: "660f1576-9451-4015-b19c-640193d50ccc") : secret "image-registry-tls" not found Sep 08 11:31:43.852926 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.852662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvc4\" (UniqueName: \"kubernetes.io/projected/a3af17fe-667d-4032-9a0e-aaadfcdacb2f-kube-api-access-7xvc4\") pod \"console-operator-85b454874b-2mxp7\" (UID: \"a3af17fe-667d-4032-9a0e-aaadfcdacb2f\") " pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:43.852926 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.852891 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/660f1576-9451-4015-b19c-640193d50ccc-trusted-ca\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.852926 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.852922 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44bd9c6-f403-43f1-820f-e49f9ded15e6-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.853124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.852945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3af17fe-667d-4032-9a0e-aaadfcdacb2f-serving-cert\") pod \"console-operator-85b454874b-2mxp7\" (UID: \"a3af17fe-667d-4032-9a0e-aaadfcdacb2f\") " pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:43.853124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.852978 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5nz2\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-kube-api-access-f5nz2\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.853752 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.853233 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f44bd9c6-f403-43f1-820f-e49f9ded15e6-tmp\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.853752 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.853259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3af17fe-667d-4032-9a0e-aaadfcdacb2f-trusted-ca\") pod \"console-operator-85b454874b-2mxp7\" (UID: \"a3af17fe-667d-4032-9a0e-aaadfcdacb2f\") " pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:43.853752 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.853286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/660f1576-9451-4015-b19c-640193d50ccc-ca-trust-extracted\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.853752 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.853307 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3af17fe-667d-4032-9a0e-aaadfcdacb2f-config\") pod \"console-operator-85b454874b-2mxp7\" (UID: \"a3af17fe-667d-4032-9a0e-aaadfcdacb2f\") " pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:43.853752 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.853338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/660f1576-9451-4015-b19c-640193d50ccc-installation-pull-secrets\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.853752 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.853353 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44bd9c6-f403-43f1-820f-e49f9ded15e6-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.853752 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.853394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f44bd9c6-f403-43f1-820f-e49f9ded15e6-serving-cert\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.853752 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.853446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/660f1576-9451-4015-b19c-640193d50ccc-registry-certificates\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.853752 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.853671 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f44bd9c6-f403-43f1-820f-e49f9ded15e6-tmp\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.853752 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.853684 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/660f1576-9451-4015-b19c-640193d50ccc-ca-trust-extracted\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.854232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.853975 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/660f1576-9451-4015-b19c-640193d50ccc-registry-certificates\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.854232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.854085 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/660f1576-9451-4015-b19c-640193d50ccc-trusted-ca\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.854232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.854091 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44bd9c6-f403-43f1-820f-e49f9ded15e6-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.854232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.853393 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f44bd9c6-f403-43f1-820f-e49f9ded15e6-snapshots\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.854583 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.854533 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3af17fe-667d-4032-9a0e-aaadfcdacb2f-trusted-ca\") pod \"console-operator-85b454874b-2mxp7\" (UID: \"a3af17fe-667d-4032-9a0e-aaadfcdacb2f\") " pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:43.854739 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.854716 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3af17fe-667d-4032-9a0e-aaadfcdacb2f-config\") pod \"console-operator-85b454874b-2mxp7\" (UID: \"a3af17fe-667d-4032-9a0e-aaadfcdacb2f\") " pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:43.856800 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.856754 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/660f1576-9451-4015-b19c-640193d50ccc-image-registry-private-configuration\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.857580 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.857557 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f44bd9c6-f403-43f1-820f-e49f9ded15e6-serving-cert\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.857744 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.857724 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3af17fe-667d-4032-9a0e-aaadfcdacb2f-serving-cert\") pod \"console-operator-85b454874b-2mxp7\" (UID: \"a3af17fe-667d-4032-9a0e-aaadfcdacb2f\") " pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:43.858199 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.858177 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/660f1576-9451-4015-b19c-640193d50ccc-installation-pull-secrets\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.862054 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.862009 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4g7t\" (UniqueName: \"kubernetes.io/projected/f44bd9c6-f403-43f1-820f-e49f9ded15e6-kube-api-access-z4g7t\") pod \"insights-operator-76bcd6ccf5-p5jkx\" (UID: \"f44bd9c6-f403-43f1-820f-e49f9ded15e6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.862720 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.862692 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5nz2\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-kube-api-access-f5nz2\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.863130 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.863079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-bound-sa-token\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:43.863407 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.863347 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xvc4\" (UniqueName: \"kubernetes.io/projected/a3af17fe-667d-4032-9a0e-aaadfcdacb2f-kube-api-access-7xvc4\") pod \"console-operator-85b454874b-2mxp7\" (UID: \"a3af17fe-667d-4032-9a0e-aaadfcdacb2f\") " pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:43.911032 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.911003 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" Sep 08 11:31:43.917242 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:43.917227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:44.075666 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:44.075484 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5"] Sep 08 11:31:44.078318 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:31:44.078293 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde336abd_d8f8_429f_ab55_0041367141c5.slice/crio-bb0383899bb8585a8d8f51e59ff597b6f28de3d2a4a8c6ed6385fc66c63a13e6 WatchSource:0}: Error finding container bb0383899bb8585a8d8f51e59ff597b6f28de3d2a4a8c6ed6385fc66c63a13e6: Status 404 returned error can't find the container with id bb0383899bb8585a8d8f51e59ff597b6f28de3d2a4a8c6ed6385fc66c63a13e6 Sep 08 11:31:44.185671 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:44.185636 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-p5jkx"] Sep 08 11:31:44.188088 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:31:44.188062 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44bd9c6_f403_43f1_820f_e49f9ded15e6.slice/crio-d0a758322786bc9e4032f04d70cc8d925462f164b5037b2647dbd81c62f55eb6 WatchSource:0}: Error finding container d0a758322786bc9e4032f04d70cc8d925462f164b5037b2647dbd81c62f55eb6: Status 404 returned error can't find the container with id d0a758322786bc9e4032f04d70cc8d925462f164b5037b2647dbd81c62f55eb6 Sep 08 11:31:44.195672 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:44.195577 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-2mxp7"] Sep 08 11:31:44.197978 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:31:44.197957 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3af17fe_667d_4032_9a0e_aaadfcdacb2f.slice/crio-b7af8e2508736e7526c38f27e4f34039edb0093a70de143b3da3f447242f0147 WatchSource:0}: Error finding container b7af8e2508736e7526c38f27e4f34039edb0093a70de143b3da3f447242f0147: Status 404 returned error can't find the container with id b7af8e2508736e7526c38f27e4f34039edb0093a70de143b3da3f447242f0147 Sep 08 11:31:44.356821 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:44.356739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:44.356966 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:44.356859 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 11:31:44.356966 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:44.356871 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-545d764448-lqfvt: secret "image-registry-tls" not found Sep 08 11:31:44.356966 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:44.356940 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls podName:660f1576-9451-4015-b19c-640193d50ccc nodeName:}" failed. No retries permitted until 2026-09-08 11:31:45.356926894 +0000 UTC m=+94.216114189 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls") pod "image-registry-545d764448-lqfvt" (UID: "660f1576-9451-4015-b19c-640193d50ccc") : secret "image-registry-tls" not found Sep 08 11:31:44.979409 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:44.979307 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" event={"ID":"f44bd9c6-f403-43f1-820f-e49f9ded15e6","Type":"ContainerStarted","Data":"d0a758322786bc9e4032f04d70cc8d925462f164b5037b2647dbd81c62f55eb6"} Sep 08 11:31:44.980957 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:44.980918 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" event={"ID":"de336abd-d8f8-429f-ab55-0041367141c5","Type":"ContainerStarted","Data":"bb0383899bb8585a8d8f51e59ff597b6f28de3d2a4a8c6ed6385fc66c63a13e6"} Sep 08 11:31:44.982266 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:44.982236 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-2mxp7" event={"ID":"a3af17fe-667d-4032-9a0e-aaadfcdacb2f","Type":"ContainerStarted","Data":"b7af8e2508736e7526c38f27e4f34039edb0093a70de143b3da3f447242f0147"} Sep 08 11:31:45.363719 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:45.363628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:45.363914 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:45.363810 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 11:31:45.363914 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:45.363832 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-545d764448-lqfvt: secret "image-registry-tls" not found Sep 08 11:31:45.363914 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:45.363905 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls podName:660f1576-9451-4015-b19c-640193d50ccc nodeName:}" failed. No retries permitted until 2026-09-08 11:31:47.363884689 +0000 UTC m=+96.223071996 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls") pod "image-registry-545d764448-lqfvt" (UID: "660f1576-9451-4015-b19c-640193d50ccc") : secret "image-registry-tls" not found Sep 08 11:31:46.990412 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:46.990317 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" event={"ID":"de336abd-d8f8-429f-ab55-0041367141c5","Type":"ContainerStarted","Data":"626d5340505db1c7b5ee258b035f2806037c78fa4b001e167b53d47b1ce87392"} Sep 08 11:31:46.992270 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:46.992248 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/0.log" Sep 08 11:31:46.992421 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:46.992290 2564 generic.go:356] "Generic (PLEG): container finished" podID="a3af17fe-667d-4032-9a0e-aaadfcdacb2f" containerID="c72847c0709644f56152f458084bcc2528092c0c62064df2f3cc56c8c564d6b8" exitCode=255 Sep 08 11:31:46.992421 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:46.992355 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-2mxp7" event={"ID":"a3af17fe-667d-4032-9a0e-aaadfcdacb2f","Type":"ContainerDied","Data":"c72847c0709644f56152f458084bcc2528092c0c62064df2f3cc56c8c564d6b8"} Sep 08 11:31:46.992606 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:46.992591 2564 scope.go:117] "RemoveContainer" containerID="c72847c0709644f56152f458084bcc2528092c0c62064df2f3cc56c8c564d6b8" Sep 08 11:31:47.009297 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:47.009246 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" podStartSLOduration=1.449099173 podStartE2EDuration="4.009224242s" totalImagesPullingTime="2.560125069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:31:47.008740045 +0000 UTC m=+95.867927383" watchObservedRunningTime="2026-09-08 11:31:47.009224242 +0000 UTC m=+95.868411562" Sep 08 11:31:47.378396 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:47.378307 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:47.378563 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:47.378537 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 11:31:47.378604 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:47.378569 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-545d764448-lqfvt: secret "image-registry-tls" not found Sep 08 11:31:47.378655 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:47.378644 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls podName:660f1576-9451-4015-b19c-640193d50ccc nodeName:}" failed. No retries permitted until 2026-09-08 11:31:51.378622061 +0000 UTC m=+100.237809360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls") pod "image-registry-545d764448-lqfvt" (UID: "660f1576-9451-4015-b19c-640193d50ccc") : secret "image-registry-tls" not found Sep 08 11:31:47.997431 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:47.997406 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 11:31:47.997878 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:47.997860 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/0.log" Sep 08 11:31:47.997953 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:47.997902 2564 generic.go:356] "Generic (PLEG): container finished" podID="a3af17fe-667d-4032-9a0e-aaadfcdacb2f" containerID="7f90c5a55776d2e59f064e4e109b4c6bfdc90d2c850fcf7f90564308a0e22ae7" exitCode=255 Sep 08 11:31:47.998016 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:47.997994 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-2mxp7" event={"ID":"a3af17fe-667d-4032-9a0e-aaadfcdacb2f","Type":"ContainerDied","Data":"7f90c5a55776d2e59f064e4e109b4c6bfdc90d2c850fcf7f90564308a0e22ae7"} Sep 08 11:31:47.998093 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:47.998077 2564 scope.go:117] "RemoveContainer" containerID="c72847c0709644f56152f458084bcc2528092c0c62064df2f3cc56c8c564d6b8" Sep 08 11:31:47.998415 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:47.998398 2564 scope.go:117] "RemoveContainer" containerID="7f90c5a55776d2e59f064e4e109b4c6bfdc90d2c850fcf7f90564308a0e22ae7" Sep 08 11:31:47.998691 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:47.998598 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-2mxp7_openshift-console-operator(a3af17fe-667d-4032-9a0e-aaadfcdacb2f)\"" pod="openshift-console-operator/console-operator-85b454874b-2mxp7" podUID="a3af17fe-667d-4032-9a0e-aaadfcdacb2f" Sep 08 11:31:48.000154 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:48.000130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" event={"ID":"f44bd9c6-f403-43f1-820f-e49f9ded15e6","Type":"ContainerStarted","Data":"e4a14522358969b367b02a7857754e14fe87c8223f3c2d94a3f84827ee4a6397"} Sep 08 11:31:48.039945 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:48.039895 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" podStartSLOduration=1.318642606 podStartE2EDuration="5.039880578s" totalImagesPullingTime="3.721237972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:31:48.039015919 +0000 UTC m=+96.898203240" watchObservedRunningTime="2026-09-08 11:31:48.039880578 +0000 UTC m=+96.899067896" Sep 08 11:31:49.004900 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.004872 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 11:31:49.005353 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.005256 2564 scope.go:117] "RemoveContainer" containerID="7f90c5a55776d2e59f064e4e109b4c6bfdc90d2c850fcf7f90564308a0e22ae7" Sep 08 11:31:49.005481 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:49.005449 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-2mxp7_openshift-console-operator(a3af17fe-667d-4032-9a0e-aaadfcdacb2f)\"" pod="openshift-console-operator/console-operator-85b454874b-2mxp7" podUID="a3af17fe-667d-4032-9a0e-aaadfcdacb2f" Sep 08 11:31:49.087566 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.087533 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pghh7"] Sep 08 11:31:49.091921 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.091901 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.094320 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.094292 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vrknd\"" Sep 08 11:31:49.094555 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.094538 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 08 11:31:49.094772 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.094759 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 08 11:31:49.101189 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.101166 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pghh7"] Sep 08 11:31:49.194588 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.194563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3830412b-f8ce-406d-b026-369258aba608-crio-socket\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.194736 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.194600 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.194736 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.194630 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3830412b-f8ce-406d-b026-369258aba608-data-volume\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.194736 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.194655 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.194736 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.194676 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46n2\" (UniqueName: \"kubernetes.io/projected/3830412b-f8ce-406d-b026-369258aba608-kube-api-access-q46n2\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.295361 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.295283 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3830412b-f8ce-406d-b026-369258aba608-crio-socket\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.295361 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.295338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.295577 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.295398 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3830412b-f8ce-406d-b026-369258aba608-data-volume\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.295577 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.295416 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3830412b-f8ce-406d-b026-369258aba608-crio-socket\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.295577 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.295440 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.295577 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.295463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46n2\" (UniqueName: \"kubernetes.io/projected/3830412b-f8ce-406d-b026-369258aba608-kube-api-access-q46n2\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.295577 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:49.295470 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 11:31:49.295577 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:49.295534 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm podName:3830412b-f8ce-406d-b026-369258aba608 nodeName:}" failed. No retries permitted until 2026-09-08 11:31:49.795520405 +0000 UTC m=+98.654707704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pghh7" (UID: "3830412b-f8ce-406d-b026-369258aba608") : configmap "kube-rbac-proxy" not found Sep 08 11:31:49.295577 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:49.295574 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 11:31:49.295837 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:49.295630 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls podName:3830412b-f8ce-406d-b026-369258aba608 nodeName:}" failed. No retries permitted until 2026-09-08 11:31:49.795615439 +0000 UTC m=+98.654802735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pghh7" (UID: "3830412b-f8ce-406d-b026-369258aba608") : secret "insights-runtime-extractor-tls" not found Sep 08 11:31:49.295837 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.295657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3830412b-f8ce-406d-b026-369258aba608-data-volume\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.305960 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.305933 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46n2\" (UniqueName: \"kubernetes.io/projected/3830412b-f8ce-406d-b026-369258aba608-kube-api-access-q46n2\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.799925 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.799885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.800088 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:49.799935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:49.800088 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:49.800007 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 11:31:49.800088 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:49.800026 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 11:31:49.800088 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:49.800065 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls podName:3830412b-f8ce-406d-b026-369258aba608 nodeName:}" failed. No retries permitted until 2026-09-08 11:31:50.800052365 +0000 UTC m=+99.659239675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pghh7" (UID: "3830412b-f8ce-406d-b026-369258aba608") : secret "insights-runtime-extractor-tls" not found Sep 08 11:31:49.800088 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:49.800079 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm podName:3830412b-f8ce-406d-b026-369258aba608 nodeName:}" failed. No retries permitted until 2026-09-08 11:31:50.800071912 +0000 UTC m=+99.659259208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pghh7" (UID: "3830412b-f8ce-406d-b026-369258aba608") : configmap "kube-rbac-proxy" not found Sep 08 11:31:50.806193 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:50.806144 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:50.806669 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:50.806272 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 11:31:50.806669 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:50.806312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:50.806669 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:50.806347 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm podName:3830412b-f8ce-406d-b026-369258aba608 nodeName:}" failed. No retries permitted until 2026-09-08 11:31:52.806327917 +0000 UTC m=+101.665515229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pghh7" (UID: "3830412b-f8ce-406d-b026-369258aba608") : configmap "kube-rbac-proxy" not found Sep 08 11:31:50.806669 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:50.806419 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 11:31:50.806669 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:50.806463 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls podName:3830412b-f8ce-406d-b026-369258aba608 nodeName:}" failed. No retries permitted until 2026-09-08 11:31:52.806451137 +0000 UTC m=+101.665638455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pghh7" (UID: "3830412b-f8ce-406d-b026-369258aba608") : secret "insights-runtime-extractor-tls" not found Sep 08 11:31:51.410447 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:51.410401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:51.410627 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:51.410572 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 11:31:51.410627 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:51.410593 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-545d764448-lqfvt: secret "image-registry-tls" not found Sep 08 11:31:51.410695 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:51.410651 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls podName:660f1576-9451-4015-b19c-640193d50ccc nodeName:}" failed. No retries permitted until 2026-09-08 11:31:59.410635052 +0000 UTC m=+108.269822347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls") pod "image-registry-545d764448-lqfvt" (UID: "660f1576-9451-4015-b19c-640193d50ccc") : secret "image-registry-tls" not found Sep 08 11:31:51.630413 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:51.630386 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5vslm_15e73439-0f50-4c2e-9155-3e18ec9e5231/dns-node-resolver/0.log" Sep 08 11:31:52.822710 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:52.822673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:52.823063 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:52.822727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:52.823063 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:52.822793 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 11:31:52.823063 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:52.822831 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 11:31:52.823063 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:52.822858 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm podName:3830412b-f8ce-406d-b026-369258aba608 nodeName:}" failed. No retries permitted until 2026-09-08 11:31:56.822842867 +0000 UTC m=+105.682030166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pghh7" (UID: "3830412b-f8ce-406d-b026-369258aba608") : configmap "kube-rbac-proxy" not found Sep 08 11:31:52.823063 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:52.822874 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls podName:3830412b-f8ce-406d-b026-369258aba608 nodeName:}" failed. No retries permitted until 2026-09-08 11:31:56.822867792 +0000 UTC m=+105.682055089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pghh7" (UID: "3830412b-f8ce-406d-b026-369258aba608") : secret "insights-runtime-extractor-tls" not found Sep 08 11:31:52.927950 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:52.927922 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gbpbr" Sep 08 11:31:53.034865 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:53.034838 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sb576_c6a08843-d61b-451e-8e36-9e949a01059d/node-ca/0.log" Sep 08 11:31:53.327528 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:53.327491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:31:53.327699 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:53.327648 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 11:31:53.327745 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:53.327709 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls podName:645f6f42-ec34-4b66-bf8e-183601ddc1af nodeName:}" failed. No retries permitted until 2026-09-08 11:32:57.327694014 +0000 UTC m=+166.186881314 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls") pod "dns-default-66f5s" (UID: "645f6f42-ec34-4b66-bf8e-183601ddc1af") : secret "dns-default-metrics-tls" not found Sep 08 11:31:53.630449 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:53.630359 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lfp5b_535465a6-a1fc-4d49-917c-6bff81183ae9/serve-healthcheck-canary/0.log" Sep 08 11:31:53.917531 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:53.917505 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:53.917938 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:53.917558 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:31:53.917938 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:53.917841 2564 scope.go:117] "RemoveContainer" containerID="7f90c5a55776d2e59f064e4e109b4c6bfdc90d2c850fcf7f90564308a0e22ae7" Sep 08 11:31:53.918004 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:53.917971 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-2mxp7_openshift-console-operator(a3af17fe-667d-4032-9a0e-aaadfcdacb2f)\"" pod="openshift-console-operator/console-operator-85b454874b-2mxp7" podUID="a3af17fe-667d-4032-9a0e-aaadfcdacb2f" Sep 08 11:31:56.854148 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:56.854109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:56.854652 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:56.854158 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:31:56.854652 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:56.854244 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 11:31:56.854652 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:56.854275 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 11:31:56.854652 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:56.854333 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm podName:3830412b-f8ce-406d-b026-369258aba608 nodeName:}" failed. No retries permitted until 2026-09-08 11:32:04.854311581 +0000 UTC m=+113.713498895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pghh7" (UID: "3830412b-f8ce-406d-b026-369258aba608") : configmap "kube-rbac-proxy" not found Sep 08 11:31:56.854652 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:31:56.854353 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls podName:3830412b-f8ce-406d-b026-369258aba608 nodeName:}" failed. No retries permitted until 2026-09-08 11:32:04.854344257 +0000 UTC m=+113.713531552 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pghh7" (UID: "3830412b-f8ce-406d-b026-369258aba608") : secret "insights-runtime-extractor-tls" not found Sep 08 11:31:59.471633 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:59.471600 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:59.473888 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:59.473856 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls\") pod \"image-registry-545d764448-lqfvt\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:59.523933 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:59.523908 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:31:59.795224 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:31:59.795152 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-545d764448-lqfvt"] Sep 08 11:31:59.797831 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:31:59.797807 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660f1576_9451_4015_b19c_640193d50ccc.slice/crio-f370f28e003f4af6466717fa3fb758e7e8ae2045f5d15c5b8b94e9cb485c5639 WatchSource:0}: Error finding container f370f28e003f4af6466717fa3fb758e7e8ae2045f5d15c5b8b94e9cb485c5639: Status 404 returned error can't find the container with id f370f28e003f4af6466717fa3fb758e7e8ae2045f5d15c5b8b94e9cb485c5639 Sep 08 11:32:00.039901 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:00.039862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-545d764448-lqfvt" event={"ID":"660f1576-9451-4015-b19c-640193d50ccc","Type":"ContainerStarted","Data":"d54e96e42c8f7841b0fc0a09c756716d161ca78acacab7d52c8aa4e2553c4c92"} Sep 08 11:32:00.040056 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:00.039920 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-545d764448-lqfvt" event={"ID":"660f1576-9451-4015-b19c-640193d50ccc","Type":"ContainerStarted","Data":"f370f28e003f4af6466717fa3fb758e7e8ae2045f5d15c5b8b94e9cb485c5639"} Sep 08 11:32:00.040056 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:00.039998 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:32:00.065745 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:00.065626 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-545d764448-lqfvt" podStartSLOduration=17.065611033 podStartE2EDuration="17.065611033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:00.064652239 +0000 UTC m=+108.923839581" watchObservedRunningTime="2026-09-08 11:32:00.065611033 +0000 UTC m=+108.924798401" Sep 08 11:32:04.912257 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:04.912218 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:32:04.912643 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:04.912271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:32:04.912751 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:04.912722 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3830412b-f8ce-406d-b026-369258aba608-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:32:04.914545 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:04.914525 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3830412b-f8ce-406d-b026-369258aba608-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pghh7\" (UID: \"3830412b-f8ce-406d-b026-369258aba608\") " pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:32:05.002291 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:05.002263 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pghh7" Sep 08 11:32:05.278526 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:05.278450 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pghh7"] Sep 08 11:32:05.281792 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:32:05.281765 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3830412b_f8ce_406d_b026_369258aba608.slice/crio-358c1c46fb0e5e4110286dc6183cf5ef0730e0cbcb2eac760ba8587a259d0b1d WatchSource:0}: Error finding container 358c1c46fb0e5e4110286dc6183cf5ef0730e0cbcb2eac760ba8587a259d0b1d: Status 404 returned error can't find the container with id 358c1c46fb0e5e4110286dc6183cf5ef0730e0cbcb2eac760ba8587a259d0b1d Sep 08 11:32:06.060845 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:06.060809 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pghh7" event={"ID":"3830412b-f8ce-406d-b026-369258aba608","Type":"ContainerStarted","Data":"ee79d5e437e8485b9ce47e1bc7a55afe3857da49da4644ac41badb81fbca7fd8"} Sep 08 11:32:06.061142 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:06.060874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pghh7" event={"ID":"3830412b-f8ce-406d-b026-369258aba608","Type":"ContainerStarted","Data":"ad58f3f5af14d71a631f60e8a0f18fcb49d045be6b9c9451b734bfb67a4ca599"} Sep 08 11:32:06.061142 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:06.060894 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pghh7" event={"ID":"3830412b-f8ce-406d-b026-369258aba608","Type":"ContainerStarted","Data":"358c1c46fb0e5e4110286dc6183cf5ef0730e0cbcb2eac760ba8587a259d0b1d"} Sep 08 11:32:06.647030 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:06.646995 2564 scope.go:117] "RemoveContainer" containerID="7f90c5a55776d2e59f064e4e109b4c6bfdc90d2c850fcf7f90564308a0e22ae7" Sep 08 11:32:07.066405 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:07.066314 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 11:32:07.066849 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:07.066402 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-2mxp7" event={"ID":"a3af17fe-667d-4032-9a0e-aaadfcdacb2f","Type":"ContainerStarted","Data":"1f8aaebf083a6e03160af48aab4c58ded15b9d8fc0d50bcb342eb4b06b83dac3"} Sep 08 11:32:07.067214 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:07.067179 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:32:07.086490 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:07.086447 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-2mxp7" podStartSLOduration=21.648786504 podStartE2EDuration="24.086433876s" totalImagesPullingTime="2.437647372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:07.085611591 +0000 UTC m=+115.944798928" watchObservedRunningTime="2026-09-08 11:32:07.086433876 +0000 UTC m=+115.945621194" Sep 08 11:32:07.668507 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:07.668478 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-2mxp7" Sep 08 11:32:08.071978 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:08.071887 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pghh7" event={"ID":"3830412b-f8ce-406d-b026-369258aba608","Type":"ContainerStarted","Data":"3c37430195481a146f2698fe77324dc93d4f5faaf450780ae2e7852da8162b1b"} Sep 08 11:32:08.092427 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:08.092383 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pghh7" podStartSLOduration=16.792053438 podStartE2EDuration="19.092353338s" totalImagesPullingTime="2.3002999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 11:31:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:08.09052427 +0000 UTC m=+116.949711608" watchObservedRunningTime="2026-09-08 11:32:08.092353338 +0000 UTC m=+116.951540655" Sep 08 11:32:15.859379 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.859344 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c54d755b4-slcc9"] Sep 08 11:32:15.861619 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.861601 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c54d755b4-slcc9" Sep 08 11:32:15.865972 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.865937 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 08 11:32:15.866076 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.866010 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 08 11:32:15.866076 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.866048 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 08 11:32:15.866163 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.866012 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-f9sw2\"" Sep 08 11:32:15.866163 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.866015 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 08 11:32:15.877941 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.877915 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c54d755b4-slcc9"] Sep 08 11:32:15.919582 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.919550 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-545d764448-lqfvt"] Sep 08 11:32:15.924107 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.924072 2564 patch_prober.go:28] interesting pod/image-registry-545d764448-lqfvt container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 08 11:32:15.924247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.924156 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-545d764448-lqfvt" podUID="660f1576-9451-4015-b19c-640193d50ccc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 11:32:15.958676 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.958648 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-wlp42"] Sep 08 11:32:15.960533 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.960517 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-wlp42" Sep 08 11:32:15.962892 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.962871 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 08 11:32:15.963453 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.963437 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-w7qww\"" Sep 08 11:32:15.963543 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.963496 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 08 11:32:15.972500 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.972480 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-wlp42"] Sep 08 11:32:15.994775 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.994752 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b092c37d-5ba4-4a1c-8e30-7698ea6990a0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c54d755b4-slcc9\" (UID: \"b092c37d-5ba4-4a1c-8e30-7698ea6990a0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c54d755b4-slcc9" Sep 08 11:32:15.994886 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:15.994820 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlfrj\" (UniqueName: \"kubernetes.io/projected/b092c37d-5ba4-4a1c-8e30-7698ea6990a0-kube-api-access-qlfrj\") pod \"managed-serviceaccount-addon-agent-7c54d755b4-slcc9\" (UID: \"b092c37d-5ba4-4a1c-8e30-7698ea6990a0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c54d755b4-slcc9" Sep 08 11:32:16.095722 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.095684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b092c37d-5ba4-4a1c-8e30-7698ea6990a0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c54d755b4-slcc9\" (UID: \"b092c37d-5ba4-4a1c-8e30-7698ea6990a0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c54d755b4-slcc9" Sep 08 11:32:16.095906 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.095744 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7twbr\" (UniqueName: \"kubernetes.io/projected/8bb30630-55f6-4d76-9406-958da86179d3-kube-api-access-7twbr\") pod \"downloads-85d98c998f-wlp42\" (UID: \"8bb30630-55f6-4d76-9406-958da86179d3\") " pod="openshift-console/downloads-85d98c998f-wlp42" Sep 08 11:32:16.095906 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.095800 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlfrj\" (UniqueName: \"kubernetes.io/projected/b092c37d-5ba4-4a1c-8e30-7698ea6990a0-kube-api-access-qlfrj\") pod \"managed-serviceaccount-addon-agent-7c54d755b4-slcc9\" (UID: \"b092c37d-5ba4-4a1c-8e30-7698ea6990a0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c54d755b4-slcc9" Sep 08 11:32:16.098937 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.098910 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b092c37d-5ba4-4a1c-8e30-7698ea6990a0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c54d755b4-slcc9\" (UID: \"b092c37d-5ba4-4a1c-8e30-7698ea6990a0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c54d755b4-slcc9" Sep 08 11:32:16.108886 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.108858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlfrj\" (UniqueName: \"kubernetes.io/projected/b092c37d-5ba4-4a1c-8e30-7698ea6990a0-kube-api-access-qlfrj\") pod \"managed-serviceaccount-addon-agent-7c54d755b4-slcc9\" (UID: \"b092c37d-5ba4-4a1c-8e30-7698ea6990a0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c54d755b4-slcc9" Sep 08 11:32:16.182304 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.182278 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c54d755b4-slcc9" Sep 08 11:32:16.196295 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.196262 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7twbr\" (UniqueName: \"kubernetes.io/projected/8bb30630-55f6-4d76-9406-958da86179d3-kube-api-access-7twbr\") pod \"downloads-85d98c998f-wlp42\" (UID: \"8bb30630-55f6-4d76-9406-958da86179d3\") " pod="openshift-console/downloads-85d98c998f-wlp42" Sep 08 11:32:16.205990 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.205964 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7twbr\" (UniqueName: \"kubernetes.io/projected/8bb30630-55f6-4d76-9406-958da86179d3-kube-api-access-7twbr\") pod \"downloads-85d98c998f-wlp42\" (UID: \"8bb30630-55f6-4d76-9406-958da86179d3\") " pod="openshift-console/downloads-85d98c998f-wlp42" Sep 08 11:32:16.269511 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.269469 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-wlp42" Sep 08 11:32:16.455921 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.455848 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c54d755b4-slcc9"] Sep 08 11:32:16.458553 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:32:16.458528 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb092c37d_5ba4_4a1c_8e30_7698ea6990a0.slice/crio-771eb969937f6563b1e49f7901bdcd25f262075fa85f1fe68150eb1e968a3152 WatchSource:0}: Error finding container 771eb969937f6563b1e49f7901bdcd25f262075fa85f1fe68150eb1e968a3152: Status 404 returned error can't find the container with id 771eb969937f6563b1e49f7901bdcd25f262075fa85f1fe68150eb1e968a3152 Sep 08 11:32:16.542896 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.542868 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-wlp42"] Sep 08 11:32:16.545986 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:32:16.545955 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb30630_55f6_4d76_9406_958da86179d3.slice/crio-47585f56bee892f5189f8088b1d03a1d689e3a20ba1fb49bf37f3170990aa499 WatchSource:0}: Error finding container 47585f56bee892f5189f8088b1d03a1d689e3a20ba1fb49bf37f3170990aa499: Status 404 returned error can't find the container with id 47585f56bee892f5189f8088b1d03a1d689e3a20ba1fb49bf37f3170990aa499 Sep 08 11:32:16.581316 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.581293 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65cddccf-s4q77"] Sep 08 11:32:16.583736 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.583720 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.586412 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.586354 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 11:32:16.586511 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.586489 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 11:32:16.586578 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.586512 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 11:32:16.586578 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.586525 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 11:32:16.586578 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.586522 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4fm94\"" Sep 08 11:32:16.586578 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.586565 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 11:32:16.598303 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.598282 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65cddccf-s4q77"] Sep 08 11:32:16.700461 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.700433 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-service-ca\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.700574 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.700475 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-serving-cert\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.700574 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.700508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr2xl\" (UniqueName: \"kubernetes.io/projected/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-kube-api-access-jr2xl\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.700656 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.700582 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-oauth-serving-cert\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.700656 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.700612 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-oauth-config\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.700656 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.700652 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-config\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.800954 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.800888 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-oauth-serving-cert\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.800954 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.800922 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-oauth-config\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.801129 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.800958 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-config\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.801129 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.801071 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-service-ca\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.801129 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.801116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-serving-cert\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.801273 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.801244 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr2xl\" (UniqueName: \"kubernetes.io/projected/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-kube-api-access-jr2xl\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.801728 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.801704 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-config\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.801830 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.801775 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-service-ca\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.801890 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.801868 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-oauth-serving-cert\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.803305 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.803282 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-oauth-config\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.803403 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.803386 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-serving-cert\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.811018 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.811001 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr2xl\" (UniqueName: \"kubernetes.io/projected/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-kube-api-access-jr2xl\") pod \"console-65cddccf-s4q77\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:16.893014 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:16.892986 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:17.104270 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:17.104199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-wlp42" event={"ID":"8bb30630-55f6-4d76-9406-958da86179d3","Type":"ContainerStarted","Data":"47585f56bee892f5189f8088b1d03a1d689e3a20ba1fb49bf37f3170990aa499"} Sep 08 11:32:17.105661 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:17.105635 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c54d755b4-slcc9" event={"ID":"b092c37d-5ba4-4a1c-8e30-7698ea6990a0","Type":"ContainerStarted","Data":"771eb969937f6563b1e49f7901bdcd25f262075fa85f1fe68150eb1e968a3152"} Sep 08 11:32:17.165437 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:17.165407 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65cddccf-s4q77"] Sep 08 11:32:17.168275 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:32:17.168248 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6978d2_b089_4dfa_bfb8_936174c3bfb6.slice/crio-ca40731f100c7d27593aec039c2d66b61f8996b61b47ef2eccc3caff5c6a7eb9 WatchSource:0}: Error finding container ca40731f100c7d27593aec039c2d66b61f8996b61b47ef2eccc3caff5c6a7eb9: Status 404 returned error can't find the container with id ca40731f100c7d27593aec039c2d66b61f8996b61b47ef2eccc3caff5c6a7eb9 Sep 08 11:32:18.115772 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:18.115734 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cddccf-s4q77" event={"ID":"1f6978d2-b089-4dfa-bfb8-936174c3bfb6","Type":"ContainerStarted","Data":"ca40731f100c7d27593aec039c2d66b61f8996b61b47ef2eccc3caff5c6a7eb9"} Sep 08 11:32:18.159106 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:18.159070 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q6n28"] Sep 08 11:32:18.163109 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:18.163083 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q6n28" Sep 08 11:32:18.167285 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:18.167263 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 08 11:32:18.167426 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:18.167287 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-s2wt6\"" Sep 08 11:32:18.171741 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:18.171683 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q6n28"] Sep 08 11:32:18.212888 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:18.212808 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3aca23dd-e56b-4e85-8057-ec729bd673e3-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-q6n28\" (UID: \"3aca23dd-e56b-4e85-8057-ec729bd673e3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q6n28" Sep 08 11:32:18.313710 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:18.313627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3aca23dd-e56b-4e85-8057-ec729bd673e3-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-q6n28\" (UID: \"3aca23dd-e56b-4e85-8057-ec729bd673e3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q6n28" Sep 08 11:32:18.313900 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:32:18.313770 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 08 11:32:18.313900 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:32:18.313842 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aca23dd-e56b-4e85-8057-ec729bd673e3-tls-certificates podName:3aca23dd-e56b-4e85-8057-ec729bd673e3 nodeName:}" failed. No retries permitted until 2026-09-08 11:32:18.813823046 +0000 UTC m=+127.673010342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/3aca23dd-e56b-4e85-8057-ec729bd673e3-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-q6n28" (UID: "3aca23dd-e56b-4e85-8057-ec729bd673e3") : secret "prometheus-operator-admission-webhook-tls" not found Sep 08 11:32:18.817487 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:18.817451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3aca23dd-e56b-4e85-8057-ec729bd673e3-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-q6n28\" (UID: \"3aca23dd-e56b-4e85-8057-ec729bd673e3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q6n28" Sep 08 11:32:18.820321 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:18.820291 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3aca23dd-e56b-4e85-8057-ec729bd673e3-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-q6n28\" (UID: \"3aca23dd-e56b-4e85-8057-ec729bd673e3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q6n28" Sep 08 11:32:19.076281 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:19.076203 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q6n28" Sep 08 11:32:20.430942 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:20.430826 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs\") pod \"network-metrics-daemon-rr4sq\" (UID: \"e3b04def-aa09-4e3b-ba69-8a8838695a83\") " pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:32:20.433644 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:20.433620 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3b04def-aa09-4e3b-ba69-8a8838695a83-metrics-certs\") pod \"network-metrics-daemon-rr4sq\" (UID: \"e3b04def-aa09-4e3b-ba69-8a8838695a83\") " pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:32:20.569842 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:20.569801 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xt4h2\"" Sep 08 11:32:20.578058 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:20.578022 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rr4sq" Sep 08 11:32:21.112485 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:21.112389 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rr4sq"] Sep 08 11:32:21.115566 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:21.115407 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q6n28"] Sep 08 11:32:21.116929 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:32:21.116892 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b04def_aa09_4e3b_ba69_8a8838695a83.slice/crio-0ccf3aea0bc5c367a98d64b2998f370d7ab61b108b8f753bb5f265c6ba77fb3d WatchSource:0}: Error finding container 0ccf3aea0bc5c367a98d64b2998f370d7ab61b108b8f753bb5f265c6ba77fb3d: Status 404 returned error can't find the container with id 0ccf3aea0bc5c367a98d64b2998f370d7ab61b108b8f753bb5f265c6ba77fb3d Sep 08 11:32:21.118409 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:32:21.118385 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aca23dd_e56b_4e85_8057_ec729bd673e3.slice/crio-55e41b94c2c3cff6b67da8ecfdb457e8546c69c5efa5c5a882fe5401c417927f WatchSource:0}: Error finding container 55e41b94c2c3cff6b67da8ecfdb457e8546c69c5efa5c5a882fe5401c417927f: Status 404 returned error can't find the container with id 55e41b94c2c3cff6b67da8ecfdb457e8546c69c5efa5c5a882fe5401c417927f Sep 08 11:32:21.130869 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:21.130822 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c54d755b4-slcc9" event={"ID":"b092c37d-5ba4-4a1c-8e30-7698ea6990a0","Type":"ContainerStarted","Data":"6204a240128e130aeb5d5ac6744e9701d5b3c308db8ae00aa9ba885cd3c33c0a"} Sep 08 11:32:21.132862 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:21.132815 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cddccf-s4q77" event={"ID":"1f6978d2-b089-4dfa-bfb8-936174c3bfb6","Type":"ContainerStarted","Data":"ee33c75e96f48333c4f1fcfba9c3de822a97673686ca5d56080c19df867a561a"} Sep 08 11:32:21.134215 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:21.134190 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q6n28" event={"ID":"3aca23dd-e56b-4e85-8057-ec729bd673e3","Type":"ContainerStarted","Data":"55e41b94c2c3cff6b67da8ecfdb457e8546c69c5efa5c5a882fe5401c417927f"} Sep 08 11:32:21.135612 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:21.135586 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rr4sq" event={"ID":"e3b04def-aa09-4e3b-ba69-8a8838695a83","Type":"ContainerStarted","Data":"0ccf3aea0bc5c367a98d64b2998f370d7ab61b108b8f753bb5f265c6ba77fb3d"} Sep 08 11:32:21.147000 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:21.146951 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c54d755b4-slcc9" podStartSLOduration=1.767688942 podStartE2EDuration="6.146932822s" totalImagesPullingTime="4.37924388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:21.146261175 +0000 UTC m=+130.005448493" watchObservedRunningTime="2026-09-08 11:32:21.146932822 +0000 UTC m=+130.006120141" Sep 08 11:32:21.178146 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:21.177938 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65cddccf-s4q77" podStartSLOduration=1.502107231 podStartE2EDuration="5.177918484s" totalImagesPullingTime="3.675811253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:21.177857234 +0000 UTC m=+130.037044564" watchObservedRunningTime="2026-09-08 11:32:21.177918484 +0000 UTC m=+130.037105793" Sep 08 11:32:23.146027 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:23.145979 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q6n28" event={"ID":"3aca23dd-e56b-4e85-8057-ec729bd673e3","Type":"ContainerStarted","Data":"fdb3ed6007015add93aeb96c5385d372d339d2af5c6a9672f7c65a3ee18df986"} Sep 08 11:32:23.146531 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:23.146092 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q6n28" Sep 08 11:32:23.148815 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:23.148779 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rr4sq" event={"ID":"e3b04def-aa09-4e3b-ba69-8a8838695a83","Type":"ContainerStarted","Data":"76a0506b79561ab9b3171b30d0464f4f2a2047a880aa0da27192da3d46bba6c5"} Sep 08 11:32:23.148943 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:23.148848 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rr4sq" event={"ID":"e3b04def-aa09-4e3b-ba69-8a8838695a83","Type":"ContainerStarted","Data":"9045e8914ebfdb9c081d404b12ac86122e09ed3a39e8dc59051bfe67e5971d28"} Sep 08 11:32:23.152395 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:23.152354 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q6n28" Sep 08 11:32:23.163293 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:23.163242 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-q6n28" podStartSLOduration=3.883882319 podStartE2EDuration="5.163226142s" totalImagesPullingTime="1.279343823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:23.163142681 +0000 UTC m=+132.022330019" watchObservedRunningTime="2026-09-08 11:32:23.163226142 +0000 UTC m=+132.022413483" Sep 08 11:32:23.179784 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:23.179744 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rr4sq" podStartSLOduration=130.890831409 podStartE2EDuration="2m12.179730972s" totalImagesPullingTime="1.288899563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:23.179355135 +0000 UTC m=+132.038542474" watchObservedRunningTime="2026-09-08 11:32:23.179730972 +0000 UTC m=+132.038918291" Sep 08 11:32:24.221068 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.221033 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-r568v"] Sep 08 11:32:24.224413 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.224384 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" Sep 08 11:32:24.226877 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.226849 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 08 11:32:24.228256 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.228078 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 08 11:32:24.228256 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.228213 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 08 11:32:24.228454 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.228316 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 08 11:32:24.228454 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.228213 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 08 11:32:24.228568 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.228215 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-42zp8\"" Sep 08 11:32:24.233207 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.233149 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-r568v"] Sep 08 11:32:24.369812 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.369782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/db55a245-eaef-4e08-9bc4-b09356c8db5f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-r568v\" (UID: \"db55a245-eaef-4e08-9bc4-b09356c8db5f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" Sep 08 11:32:24.369812 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.369818 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qx4p\" (UniqueName: \"kubernetes.io/projected/db55a245-eaef-4e08-9bc4-b09356c8db5f-kube-api-access-8qx4p\") pod \"prometheus-operator-6f4946db56-r568v\" (UID: \"db55a245-eaef-4e08-9bc4-b09356c8db5f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" Sep 08 11:32:24.370033 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.369850 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db55a245-eaef-4e08-9bc4-b09356c8db5f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-r568v\" (UID: \"db55a245-eaef-4e08-9bc4-b09356c8db5f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" Sep 08 11:32:24.370033 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.369877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db55a245-eaef-4e08-9bc4-b09356c8db5f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-r568v\" (UID: \"db55a245-eaef-4e08-9bc4-b09356c8db5f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" Sep 08 11:32:24.471329 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.471248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/db55a245-eaef-4e08-9bc4-b09356c8db5f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-r568v\" (UID: \"db55a245-eaef-4e08-9bc4-b09356c8db5f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" Sep 08 11:32:24.471329 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.471296 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qx4p\" (UniqueName: \"kubernetes.io/projected/db55a245-eaef-4e08-9bc4-b09356c8db5f-kube-api-access-8qx4p\") pod \"prometheus-operator-6f4946db56-r568v\" (UID: \"db55a245-eaef-4e08-9bc4-b09356c8db5f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" Sep 08 11:32:24.471561 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.471333 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db55a245-eaef-4e08-9bc4-b09356c8db5f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-r568v\" (UID: \"db55a245-eaef-4e08-9bc4-b09356c8db5f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" Sep 08 11:32:24.471561 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.471362 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db55a245-eaef-4e08-9bc4-b09356c8db5f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-r568v\" (UID: \"db55a245-eaef-4e08-9bc4-b09356c8db5f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" Sep 08 11:32:24.472097 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.472072 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db55a245-eaef-4e08-9bc4-b09356c8db5f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-r568v\" (UID: \"db55a245-eaef-4e08-9bc4-b09356c8db5f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" Sep 08 11:32:24.473845 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.473812 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/db55a245-eaef-4e08-9bc4-b09356c8db5f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-r568v\" (UID: \"db55a245-eaef-4e08-9bc4-b09356c8db5f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" Sep 08 11:32:24.473979 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.473961 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db55a245-eaef-4e08-9bc4-b09356c8db5f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-r568v\" (UID: \"db55a245-eaef-4e08-9bc4-b09356c8db5f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" Sep 08 11:32:24.491046 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.491020 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qx4p\" (UniqueName: \"kubernetes.io/projected/db55a245-eaef-4e08-9bc4-b09356c8db5f-kube-api-access-8qx4p\") pod \"prometheus-operator-6f4946db56-r568v\" (UID: \"db55a245-eaef-4e08-9bc4-b09356c8db5f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" Sep 08 11:32:24.535527 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.535495 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" Sep 08 11:32:24.815614 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:24.815585 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-r568v"] Sep 08 11:32:24.818330 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:32:24.818299 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb55a245_eaef_4e08_9bc4_b09356c8db5f.slice/crio-eb86042a6623239a40af7f12ec63b7761d8cc30d3122fb91a17e2bdaa89b1276 WatchSource:0}: Error finding container eb86042a6623239a40af7f12ec63b7761d8cc30d3122fb91a17e2bdaa89b1276: Status 404 returned error can't find the container with id eb86042a6623239a40af7f12ec63b7761d8cc30d3122fb91a17e2bdaa89b1276 Sep 08 11:32:25.157653 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:25.157614 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" event={"ID":"db55a245-eaef-4e08-9bc4-b09356c8db5f","Type":"ContainerStarted","Data":"eb86042a6623239a40af7f12ec63b7761d8cc30d3122fb91a17e2bdaa89b1276"} Sep 08 11:32:25.925985 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:25.925954 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:32:26.163741 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:26.163711 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" event={"ID":"db55a245-eaef-4e08-9bc4-b09356c8db5f","Type":"ContainerStarted","Data":"053b74e3ad49fb7fd3c6f89aa31e1632f1dd5c12416a062528d2295578f56439"} Sep 08 11:32:26.893670 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:26.893630 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:26.893860 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:26.893707 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:26.899497 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:26.899469 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:27.174649 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:27.174559 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" event={"ID":"db55a245-eaef-4e08-9bc4-b09356c8db5f","Type":"ContainerStarted","Data":"897acf0080c9bd16ec213d096b6495c2037c95df8a5ffa89f25827de656e4558"} Sep 08 11:32:27.178997 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:27.178970 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:32:27.195063 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:27.195016 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-r568v" podStartSLOduration=1.969248301 podStartE2EDuration="3.194996234s" totalImagesPullingTime="1.225747933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:27.193349774 +0000 UTC m=+136.052537121" watchObservedRunningTime="2026-09-08 11:32:27.194996234 +0000 UTC m=+136.054183554" Sep 08 11:32:29.624771 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.624731 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x4fkq"] Sep 08 11:32:29.629251 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.629225 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.631762 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.631725 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 08 11:32:29.631902 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.631822 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 08 11:32:29.631902 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.631852 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 08 11:32:29.632154 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.632133 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k7x8b\"" Sep 08 11:32:29.721527 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.721498 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-metrics-client-ca\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.721527 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.721536 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-sys\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.721766 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.721560 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-node-exporter-tls\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.721766 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.721648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-node-exporter-accelerators-collector-config\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.721766 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.721703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-root\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.721766 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.721728 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.721933 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.721769 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-node-exporter-textfile\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.721933 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.721813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-node-exporter-wtmp\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.721933 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.721886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7fcr\" (UniqueName: \"kubernetes.io/projected/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-kube-api-access-b7fcr\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.822568 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.822532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-root\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.822718 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.822579 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.822718 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.822615 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-node-exporter-textfile\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.822718 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.822638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-node-exporter-wtmp\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.822718 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.822700 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7fcr\" (UniqueName: \"kubernetes.io/projected/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-kube-api-access-b7fcr\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.822880 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.822746 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-metrics-client-ca\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.822880 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.822770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-sys\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.822880 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.822804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-node-exporter-tls\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.822880 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.822832 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-node-exporter-accelerators-collector-config\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.823466 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.823439 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-node-exporter-accelerators-collector-config\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.823595 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.823521 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-root\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.823804 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.823667 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-sys\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.823804 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.823758 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-node-exporter-wtmp\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.824094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.824071 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-node-exporter-textfile\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.824223 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.824145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-metrics-client-ca\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.826303 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.826273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.826491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.826447 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-node-exporter-tls\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.836225 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.836157 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7fcr\" (UniqueName: \"kubernetes.io/projected/4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b-kube-api-access-b7fcr\") pod \"node-exporter-x4fkq\" (UID: \"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b\") " pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:29.941128 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:29.941094 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x4fkq" Sep 08 11:32:32.744696 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.744651 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p"] Sep 08 11:32:32.748886 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.748855 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.751749 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.751723 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2468quebg5pp0\"" Sep 08 11:32:32.751893 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.751801 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 08 11:32:32.751893 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.751801 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 08 11:32:32.751893 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.751833 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 08 11:32:32.752047 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.751904 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ntr8c\"" Sep 08 11:32:32.752047 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.751944 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 08 11:32:32.752186 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.752157 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 08 11:32:32.761636 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.761615 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p"] Sep 08 11:32:32.849479 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.849444 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-282w8\" (UniqueName: \"kubernetes.io/projected/63238fe1-9071-4c9d-a037-c25a0bd5560c-kube-api-access-282w8\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.849479 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.849481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-thanos-querier-tls\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.849726 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.849512 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.849726 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.849653 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.849835 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.849780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.849890 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.849843 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-grpc-tls\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.849890 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.849883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63238fe1-9071-4c9d-a037-c25a0bd5560c-metrics-client-ca\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.849981 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.849913 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.951305 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.951267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.951474 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.951438 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.951543 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.951492 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-grpc-tls\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.951543 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.951522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63238fe1-9071-4c9d-a037-c25a0bd5560c-metrics-client-ca\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.951646 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.951550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.951646 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.951582 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-282w8\" (UniqueName: \"kubernetes.io/projected/63238fe1-9071-4c9d-a037-c25a0bd5560c-kube-api-access-282w8\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.951646 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.951614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-thanos-querier-tls\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.951781 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.951651 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.952359 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.952319 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63238fe1-9071-4c9d-a037-c25a0bd5560c-metrics-client-ca\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.954567 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.954539 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.954675 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.954656 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-grpc-tls\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.954864 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.954838 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-thanos-querier-tls\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.955237 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.955193 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.955384 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.955297 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.955457 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.955392 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63238fe1-9071-4c9d-a037-c25a0bd5560c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:32.961516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:32.961491 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-282w8\" (UniqueName: \"kubernetes.io/projected/63238fe1-9071-4c9d-a037-c25a0bd5560c-kube-api-access-282w8\") pod \"thanos-querier-6d48c6b94d-s4q8p\" (UID: \"63238fe1-9071-4c9d-a037-c25a0bd5560c\") " pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:33.061299 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:33.061217 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:33.276786 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:32:33.276746 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4492f8b8_3d1e_4aaf_9e56_ae877dbffd4b.slice/crio-c76255ac9ada00cdb51b733d2dd1056d1cd724e7e8be9056fbf1e07ad66fae72 WatchSource:0}: Error finding container c76255ac9ada00cdb51b733d2dd1056d1cd724e7e8be9056fbf1e07ad66fae72: Status 404 returned error can't find the container with id c76255ac9ada00cdb51b733d2dd1056d1cd724e7e8be9056fbf1e07ad66fae72 Sep 08 11:32:33.549653 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:33.549624 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p"] Sep 08 11:32:33.551856 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:32:33.551822 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63238fe1_9071_4c9d_a037_c25a0bd5560c.slice/crio-8da11e1c4844a4d8dfc313f0cf7352fe016a7ac25d4cc350898b143b7292301d WatchSource:0}: Error finding container 8da11e1c4844a4d8dfc313f0cf7352fe016a7ac25d4cc350898b143b7292301d: Status 404 returned error can't find the container with id 8da11e1c4844a4d8dfc313f0cf7352fe016a7ac25d4cc350898b143b7292301d Sep 08 11:32:33.922767 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:33.922731 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6dd97df8-b75x6"] Sep 08 11:32:33.927517 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:33.927490 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:33.930193 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:33.930167 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 08 11:32:33.930326 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:33.930214 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 08 11:32:33.931069 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:33.931024 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 08 11:32:33.931069 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:33.931025 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fl26v\"" Sep 08 11:32:33.931334 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:33.931318 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2q2hfg9k2u1oe\"" Sep 08 11:32:33.931444 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:33.931417 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 08 11:32:33.940066 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:33.940043 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dd97df8-b75x6"] Sep 08 11:32:34.062949 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.062907 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/89e99fa4-2f32-4165-883d-228ff3a4e15b-secret-metrics-server-client-certs\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.062949 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.062953 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e99fa4-2f32-4165-883d-228ff3a4e15b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.063203 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.062986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/89e99fa4-2f32-4165-883d-228ff3a4e15b-secret-metrics-server-tls\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.063266 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.063194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/89e99fa4-2f32-4165-883d-228ff3a4e15b-metrics-server-audit-profiles\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.063266 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.063235 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/89e99fa4-2f32-4165-883d-228ff3a4e15b-client-ca-bundle\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.063397 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.063357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/89e99fa4-2f32-4165-883d-228ff3a4e15b-audit-log\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.063458 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.063410 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88q8f\" (UniqueName: \"kubernetes.io/projected/89e99fa4-2f32-4165-883d-228ff3a4e15b-kube-api-access-88q8f\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.164665 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.164563 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/89e99fa4-2f32-4165-883d-228ff3a4e15b-secret-metrics-server-client-certs\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.164665 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.164617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e99fa4-2f32-4165-883d-228ff3a4e15b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.164912 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.164677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/89e99fa4-2f32-4165-883d-228ff3a4e15b-secret-metrics-server-tls\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.164912 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.164704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/89e99fa4-2f32-4165-883d-228ff3a4e15b-metrics-server-audit-profiles\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.164912 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.164730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/89e99fa4-2f32-4165-883d-228ff3a4e15b-client-ca-bundle\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.164912 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.164813 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/89e99fa4-2f32-4165-883d-228ff3a4e15b-audit-log\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.164912 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.164837 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88q8f\" (UniqueName: \"kubernetes.io/projected/89e99fa4-2f32-4165-883d-228ff3a4e15b-kube-api-access-88q8f\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.167100 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.166441 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89e99fa4-2f32-4165-883d-228ff3a4e15b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.167100 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.166443 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/89e99fa4-2f32-4165-883d-228ff3a4e15b-metrics-server-audit-profiles\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.167100 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.166790 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/89e99fa4-2f32-4165-883d-228ff3a4e15b-audit-log\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.168440 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.168397 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/89e99fa4-2f32-4165-883d-228ff3a4e15b-secret-metrics-server-client-certs\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.169843 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.169795 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/89e99fa4-2f32-4165-883d-228ff3a4e15b-client-ca-bundle\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.170929 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.170884 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/89e99fa4-2f32-4165-883d-228ff3a4e15b-secret-metrics-server-tls\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.180570 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.180481 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88q8f\" (UniqueName: \"kubernetes.io/projected/89e99fa4-2f32-4165-883d-228ff3a4e15b-kube-api-access-88q8f\") pod \"metrics-server-6dd97df8-b75x6\" (UID: \"89e99fa4-2f32-4165-883d-228ff3a4e15b\") " pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.205639 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.205603 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x4fkq" event={"ID":"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b","Type":"ContainerStarted","Data":"c76255ac9ada00cdb51b733d2dd1056d1cd724e7e8be9056fbf1e07ad66fae72"} Sep 08 11:32:34.209657 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.209631 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-wlp42" event={"ID":"8bb30630-55f6-4d76-9406-958da86179d3","Type":"ContainerStarted","Data":"166fd0dcaa689a03d2316a22f084487b23f109a3fa5b1655addd5fdad8e90642"} Sep 08 11:32:34.209786 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.209768 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-wlp42" Sep 08 11:32:34.211700 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.211632 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" event={"ID":"63238fe1-9071-4c9d-a037-c25a0bd5560c","Type":"ContainerStarted","Data":"8da11e1c4844a4d8dfc313f0cf7352fe016a7ac25d4cc350898b143b7292301d"} Sep 08 11:32:34.222544 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.222522 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-wlp42" Sep 08 11:32:34.235171 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.234743 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-wlp42" podStartSLOduration=2.411569637 podStartE2EDuration="19.234725577s" totalImagesPullingTime="16.82315594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:34.232782955 +0000 UTC m=+143.091970298" watchObservedRunningTime="2026-09-08 11:32:34.234725577 +0000 UTC m=+143.093912896" Sep 08 11:32:34.240471 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.240122 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:34.392286 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.392248 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8"] Sep 08 11:32:34.398078 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.398021 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8" Sep 08 11:32:34.401936 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.401901 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 08 11:32:34.402216 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.402194 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vhrpv\"" Sep 08 11:32:34.405765 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.405734 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8"] Sep 08 11:32:34.453414 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.453189 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bf4f4cdb7-l6r9s"] Sep 08 11:32:34.457437 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.457395 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.465690 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.465039 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 08 11:32:34.469418 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.469343 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf4f4cdb7-l6r9s"] Sep 08 11:32:34.559256 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.559223 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dd97df8-b75x6"] Sep 08 11:32:34.563561 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:32:34.563530 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e99fa4_2f32_4165_883d_228ff3a4e15b.slice/crio-49889d869c8e7bec6551adb69a4e65d74d2eb78e8382214e590975258439dfff WatchSource:0}: Error finding container 49889d869c8e7bec6551adb69a4e65d74d2eb78e8382214e590975258439dfff: Status 404 returned error can't find the container with id 49889d869c8e7bec6551adb69a4e65d74d2eb78e8382214e590975258439dfff Sep 08 11:32:34.573569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.573106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-config\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.573569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.573163 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-serving-cert\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.573569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.573190 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-trusted-ca-bundle\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.573569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.573215 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/857c1344-c926-43c7-961b-39b71303a7df-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-75bh8\" (UID: \"857c1344-c926-43c7-961b-39b71303a7df\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8" Sep 08 11:32:34.573569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.573243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcn4q\" (UniqueName: \"kubernetes.io/projected/b535cef7-8dba-4819-b9a9-90d1c6319c19-kube-api-access-gcn4q\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.573569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.573281 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-service-ca\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.573569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.573320 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-oauth-serving-cert\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.573569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.573385 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-oauth-config\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.573569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.573413 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p4q7\" (UniqueName: \"kubernetes.io/projected/857c1344-c926-43c7-961b-39b71303a7df-kube-api-access-9p4q7\") pod \"monitoring-plugin-85bbbbbf4-75bh8\" (UID: \"857c1344-c926-43c7-961b-39b71303a7df\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8" Sep 08 11:32:34.674012 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.673872 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-config\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.674012 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.673936 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-serving-cert\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.674684 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.674651 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-trusted-ca-bundle\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.674804 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.674697 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/857c1344-c926-43c7-961b-39b71303a7df-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-75bh8\" (UID: \"857c1344-c926-43c7-961b-39b71303a7df\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8" Sep 08 11:32:34.674804 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.674733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcn4q\" (UniqueName: \"kubernetes.io/projected/b535cef7-8dba-4819-b9a9-90d1c6319c19-kube-api-access-gcn4q\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.674804 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.674780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-service-ca\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.674957 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.674820 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-config\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.674957 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.674832 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-oauth-serving-cert\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.674957 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.674909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-oauth-config\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.674957 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.674934 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p4q7\" (UniqueName: \"kubernetes.io/projected/857c1344-c926-43c7-961b-39b71303a7df-kube-api-access-9p4q7\") pod \"monitoring-plugin-85bbbbbf4-75bh8\" (UID: \"857c1344-c926-43c7-961b-39b71303a7df\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8" Sep 08 11:32:34.675484 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.675454 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-service-ca\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.675593 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.675581 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-trusted-ca-bundle\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.676110 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.676082 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-oauth-serving-cert\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.678289 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.678242 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/857c1344-c926-43c7-961b-39b71303a7df-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-75bh8\" (UID: \"857c1344-c926-43c7-961b-39b71303a7df\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8" Sep 08 11:32:34.678670 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.678646 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-serving-cert\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.679099 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.679057 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-oauth-config\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.685681 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.685655 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p4q7\" (UniqueName: \"kubernetes.io/projected/857c1344-c926-43c7-961b-39b71303a7df-kube-api-access-9p4q7\") pod \"monitoring-plugin-85bbbbbf4-75bh8\" (UID: \"857c1344-c926-43c7-961b-39b71303a7df\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8" Sep 08 11:32:34.685681 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.685678 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcn4q\" (UniqueName: \"kubernetes.io/projected/b535cef7-8dba-4819-b9a9-90d1c6319c19-kube-api-access-gcn4q\") pod \"console-7bf4f4cdb7-l6r9s\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:34.717496 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.717427 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8" Sep 08 11:32:34.773350 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:34.772230 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:35.021495 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:35.021252 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8"] Sep 08 11:32:35.117710 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:35.116284 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf4f4cdb7-l6r9s"] Sep 08 11:32:35.221464 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:35.221428 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" event={"ID":"89e99fa4-2f32-4165-883d-228ff3a4e15b","Type":"ContainerStarted","Data":"49889d869c8e7bec6551adb69a4e65d74d2eb78e8382214e590975258439dfff"} Sep 08 11:32:36.075942 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:32:36.075904 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb535cef7_8dba_4819_b9a9_90d1c6319c19.slice/crio-b62b5568e68ee40de0d05cf30e5595b9eb62d497cbfc2ba46148182fd14350fb WatchSource:0}: Error finding container b62b5568e68ee40de0d05cf30e5595b9eb62d497cbfc2ba46148182fd14350fb: Status 404 returned error can't find the container with id b62b5568e68ee40de0d05cf30e5595b9eb62d497cbfc2ba46148182fd14350fb Sep 08 11:32:36.076844 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:32:36.076813 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857c1344_c926_43c7_961b_39b71303a7df.slice/crio-1cab58850649b0ce1370b2d210725a6e3def5d08e57f2faa1706b49971c349e7 WatchSource:0}: Error finding container 1cab58850649b0ce1370b2d210725a6e3def5d08e57f2faa1706b49971c349e7: Status 404 returned error can't find the container with id 1cab58850649b0ce1370b2d210725a6e3def5d08e57f2faa1706b49971c349e7 Sep 08 11:32:36.226881 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:36.226688 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8" event={"ID":"857c1344-c926-43c7-961b-39b71303a7df","Type":"ContainerStarted","Data":"1cab58850649b0ce1370b2d210725a6e3def5d08e57f2faa1706b49971c349e7"} Sep 08 11:32:36.228341 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:36.228272 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf4f4cdb7-l6r9s" event={"ID":"b535cef7-8dba-4819-b9a9-90d1c6319c19","Type":"ContainerStarted","Data":"b62b5568e68ee40de0d05cf30e5595b9eb62d497cbfc2ba46148182fd14350fb"} Sep 08 11:32:38.241067 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:38.240989 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" event={"ID":"63238fe1-9071-4c9d-a037-c25a0bd5560c","Type":"ContainerStarted","Data":"13640692612e6571f6a794fe53fbe029af5914604efb393a93cd2f72e266f9ca"} Sep 08 11:32:38.241528 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:38.241079 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" event={"ID":"63238fe1-9071-4c9d-a037-c25a0bd5560c","Type":"ContainerStarted","Data":"3cadd7647037940580aaf6fb082bd26ccb7225bea654e1283e9323e8e3a6e702"} Sep 08 11:32:38.243192 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:38.243154 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" event={"ID":"89e99fa4-2f32-4165-883d-228ff3a4e15b","Type":"ContainerStarted","Data":"420b30c8ec34dbf7b07f5fdffedfb5e73e9f9b0f308869576492352298eb5515"} Sep 08 11:32:38.245589 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:38.245538 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf4f4cdb7-l6r9s" event={"ID":"b535cef7-8dba-4819-b9a9-90d1c6319c19","Type":"ContainerStarted","Data":"22208e87ef9603122afdf4ea438ea6e56ee09320eb016d1177e81ae6ec93f85a"} Sep 08 11:32:38.247941 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:38.247719 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x4fkq" event={"ID":"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b","Type":"ContainerStarted","Data":"fb4b46bc8dce226bc3040d37b1f76f972775793e35739b242f6b6d0fc64e0bf0"} Sep 08 11:32:38.265173 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:38.265127 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" podStartSLOduration=2.578718128 podStartE2EDuration="5.265111772s" totalImagesPullingTime="2.686393644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:38.263625168 +0000 UTC m=+147.122812497" watchObservedRunningTime="2026-09-08 11:32:38.265111772 +0000 UTC m=+147.124299090" Sep 08 11:32:38.302622 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:38.302575 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bf4f4cdb7-l6r9s" podStartSLOduration=4.302554712 podStartE2EDuration="4.302554712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:38.302065139 +0000 UTC m=+147.161252469" watchObservedRunningTime="2026-09-08 11:32:38.302554712 +0000 UTC m=+147.161742034" Sep 08 11:32:39.185004 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:39.184964 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf4f4cdb7-l6r9s"] Sep 08 11:32:39.255969 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:39.255926 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" event={"ID":"63238fe1-9071-4c9d-a037-c25a0bd5560c","Type":"ContainerStarted","Data":"25f74e402ea462c22309a0e0ec7b28e49ffd98c1f2a0c42526ed7905314154d6"} Sep 08 11:32:39.257988 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:39.257833 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8" event={"ID":"857c1344-c926-43c7-961b-39b71303a7df","Type":"ContainerStarted","Data":"66e435e22818fa21b903679d61c1812001857a4381b1f36ac56f6458f8ff82bd"} Sep 08 11:32:39.258123 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:39.258044 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8" Sep 08 11:32:39.259844 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:39.259746 2564 generic.go:356] "Generic (PLEG): container finished" podID="4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b" containerID="fb4b46bc8dce226bc3040d37b1f76f972775793e35739b242f6b6d0fc64e0bf0" exitCode=0 Sep 08 11:32:39.259844 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:39.259827 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x4fkq" event={"ID":"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b","Type":"ContainerDied","Data":"fb4b46bc8dce226bc3040d37b1f76f972775793e35739b242f6b6d0fc64e0bf0"} Sep 08 11:32:39.263424 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:39.263394 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8" Sep 08 11:32:39.281871 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:39.281817 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-75bh8" podStartSLOduration=2.767984644 podStartE2EDuration="5.281798849s" totalImagesPullingTime="2.513814205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:39.281457609 +0000 UTC m=+148.140644950" watchObservedRunningTime="2026-09-08 11:32:39.281798849 +0000 UTC m=+148.140986168" Sep 08 11:32:40.269781 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:40.269736 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x4fkq" event={"ID":"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b","Type":"ContainerStarted","Data":"af4e74827d035d299921ecebee46cbd546479a20074eb82014f0399cee43e226"} Sep 08 11:32:40.270234 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:40.269843 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x4fkq" event={"ID":"4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b","Type":"ContainerStarted","Data":"834375b8f36f3bcec98270b5445beef523fae6dbac782a4f36011ad22dd90af2"} Sep 08 11:32:40.273458 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:40.273430 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" event={"ID":"63238fe1-9071-4c9d-a037-c25a0bd5560c","Type":"ContainerStarted","Data":"1d34e20cb577a9b52990d79f90f964d9ae48ac37354829339939bf90f911650b"} Sep 08 11:32:40.273587 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:40.273495 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" event={"ID":"63238fe1-9071-4c9d-a037-c25a0bd5560c","Type":"ContainerStarted","Data":"791026ce6075297b299d11ea423c64505ea3f567b13ac21bafa1c7bc1fb2d2d7"} Sep 08 11:32:40.273587 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:40.273530 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" event={"ID":"63238fe1-9071-4c9d-a037-c25a0bd5560c","Type":"ContainerStarted","Data":"8d40399cb59084933f93572fcb587fc88d0af2f6888dbc4a58117bd7c85e1316"} Sep 08 11:32:40.292016 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:40.291349 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x4fkq" podStartSLOduration=6.899615735 podStartE2EDuration="11.291336523s" totalImagesPullingTime="4.308340424s" totalInitContainerRuntime="83.380364ms" isStatefulPod=true podCreationTimestamp="2026-09-08 11:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:40.291000724 +0000 UTC m=+149.150188062" watchObservedRunningTime="2026-09-08 11:32:40.291336523 +0000 UTC m=+149.150523841" Sep 08 11:32:40.315224 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:40.315133 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" podStartSLOduration=3.140835352 podStartE2EDuration="8.315117791s" totalImagesPullingTime="5.174282439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:32:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:32:40.313852101 +0000 UTC m=+149.173039444" watchObservedRunningTime="2026-09-08 11:32:40.315117791 +0000 UTC m=+149.174305109" Sep 08 11:32:40.939312 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:40.939270 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-545d764448-lqfvt" podUID="660f1576-9451-4015-b19c-640193d50ccc" containerName="registry" containerID="cri-o://d54e96e42c8f7841b0fc0a09c756716d161ca78acacab7d52c8aa4e2553c4c92" gracePeriod=30 Sep 08 11:32:41.238296 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.238265 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:32:41.280540 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.280503 2564 generic.go:356] "Generic (PLEG): container finished" podID="660f1576-9451-4015-b19c-640193d50ccc" containerID="d54e96e42c8f7841b0fc0a09c756716d161ca78acacab7d52c8aa4e2553c4c92" exitCode=0 Sep 08 11:32:41.280962 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.280561 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-545d764448-lqfvt" Sep 08 11:32:41.280962 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.280592 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-545d764448-lqfvt" event={"ID":"660f1576-9451-4015-b19c-640193d50ccc","Type":"ContainerDied","Data":"d54e96e42c8f7841b0fc0a09c756716d161ca78acacab7d52c8aa4e2553c4c92"} Sep 08 11:32:41.280962 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.280651 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-545d764448-lqfvt" event={"ID":"660f1576-9451-4015-b19c-640193d50ccc","Type":"ContainerDied","Data":"f370f28e003f4af6466717fa3fb758e7e8ae2045f5d15c5b8b94e9cb485c5639"} Sep 08 11:32:41.280962 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.280683 2564 scope.go:117] "RemoveContainer" containerID="d54e96e42c8f7841b0fc0a09c756716d161ca78acacab7d52c8aa4e2553c4c92" Sep 08 11:32:41.281660 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.281574 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:41.290266 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.290249 2564 scope.go:117] "RemoveContainer" containerID="d54e96e42c8f7841b0fc0a09c756716d161ca78acacab7d52c8aa4e2553c4c92" Sep 08 11:32:41.290612 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:32:41.290590 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d54e96e42c8f7841b0fc0a09c756716d161ca78acacab7d52c8aa4e2553c4c92\": container with ID starting with d54e96e42c8f7841b0fc0a09c756716d161ca78acacab7d52c8aa4e2553c4c92 not found: ID does not exist" containerID="d54e96e42c8f7841b0fc0a09c756716d161ca78acacab7d52c8aa4e2553c4c92" Sep 08 11:32:41.290705 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.290617 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d54e96e42c8f7841b0fc0a09c756716d161ca78acacab7d52c8aa4e2553c4c92"} err="failed to get container status \"d54e96e42c8f7841b0fc0a09c756716d161ca78acacab7d52c8aa4e2553c4c92\": rpc error: code = NotFound desc = could not find container \"d54e96e42c8f7841b0fc0a09c756716d161ca78acacab7d52c8aa4e2553c4c92\": container with ID starting with d54e96e42c8f7841b0fc0a09c756716d161ca78acacab7d52c8aa4e2553c4c92 not found: ID does not exist" Sep 08 11:32:41.430931 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.430903 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/660f1576-9451-4015-b19c-640193d50ccc-installation-pull-secrets\") pod \"660f1576-9451-4015-b19c-640193d50ccc\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " Sep 08 11:32:41.431099 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.430958 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/660f1576-9451-4015-b19c-640193d50ccc-ca-trust-extracted\") pod \"660f1576-9451-4015-b19c-640193d50ccc\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " Sep 08 11:32:41.431099 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.430992 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-bound-sa-token\") pod \"660f1576-9451-4015-b19c-640193d50ccc\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " Sep 08 11:32:41.431099 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.431014 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5nz2\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-kube-api-access-f5nz2\") pod \"660f1576-9451-4015-b19c-640193d50ccc\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " Sep 08 11:32:41.431099 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.431032 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/660f1576-9451-4015-b19c-640193d50ccc-trusted-ca\") pod \"660f1576-9451-4015-b19c-640193d50ccc\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " Sep 08 11:32:41.431099 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.431090 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/660f1576-9451-4015-b19c-640193d50ccc-image-registry-private-configuration\") pod \"660f1576-9451-4015-b19c-640193d50ccc\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " Sep 08 11:32:41.431394 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.431129 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls\") pod \"660f1576-9451-4015-b19c-640193d50ccc\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " Sep 08 11:32:41.431394 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.431179 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/660f1576-9451-4015-b19c-640193d50ccc-registry-certificates\") pod \"660f1576-9451-4015-b19c-640193d50ccc\" (UID: \"660f1576-9451-4015-b19c-640193d50ccc\") " Sep 08 11:32:41.431778 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.431723 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660f1576-9451-4015-b19c-640193d50ccc-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "660f1576-9451-4015-b19c-640193d50ccc" (UID: "660f1576-9451-4015-b19c-640193d50ccc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:32:41.432052 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.431973 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660f1576-9451-4015-b19c-640193d50ccc-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "660f1576-9451-4015-b19c-640193d50ccc" (UID: "660f1576-9451-4015-b19c-640193d50ccc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:32:41.433777 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.433666 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "660f1576-9451-4015-b19c-640193d50ccc" (UID: "660f1576-9451-4015-b19c-640193d50ccc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:32:41.433985 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.433932 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660f1576-9451-4015-b19c-640193d50ccc-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "660f1576-9451-4015-b19c-640193d50ccc" (UID: "660f1576-9451-4015-b19c-640193d50ccc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:32:41.433985 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.433938 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-kube-api-access-f5nz2" (OuterVolumeSpecName: "kube-api-access-f5nz2") pod "660f1576-9451-4015-b19c-640193d50ccc" (UID: "660f1576-9451-4015-b19c-640193d50ccc"). InnerVolumeSpecName "kube-api-access-f5nz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:32:41.434250 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.434217 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "660f1576-9451-4015-b19c-640193d50ccc" (UID: "660f1576-9451-4015-b19c-640193d50ccc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:32:41.434399 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.434350 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660f1576-9451-4015-b19c-640193d50ccc-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "660f1576-9451-4015-b19c-640193d50ccc" (UID: "660f1576-9451-4015-b19c-640193d50ccc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:32:41.442122 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.442098 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660f1576-9451-4015-b19c-640193d50ccc-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "660f1576-9451-4015-b19c-640193d50ccc" (UID: "660f1576-9451-4015-b19c-640193d50ccc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:32:41.532101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.532013 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/660f1576-9451-4015-b19c-640193d50ccc-ca-trust-extracted\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:32:41.532101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.532052 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-bound-sa-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:32:41.532101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.532062 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5nz2\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-kube-api-access-f5nz2\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:32:41.532101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.532070 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/660f1576-9451-4015-b19c-640193d50ccc-trusted-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:32:41.532101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.532080 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/660f1576-9451-4015-b19c-640193d50ccc-image-registry-private-configuration\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:32:41.532101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.532091 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/660f1576-9451-4015-b19c-640193d50ccc-registry-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:32:41.532101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.532105 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/660f1576-9451-4015-b19c-640193d50ccc-registry-certificates\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:32:41.532529 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.532119 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/660f1576-9451-4015-b19c-640193d50ccc-installation-pull-secrets\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:32:41.609491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.609457 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-545d764448-lqfvt"] Sep 08 11:32:41.614408 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.614357 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-545d764448-lqfvt"] Sep 08 11:32:41.653248 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:41.653219 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="660f1576-9451-4015-b19c-640193d50ccc" path="/var/lib/kubelet/pods/660f1576-9451-4015-b19c-640193d50ccc/volumes" Sep 08 11:32:42.291901 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:42.291873 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6d48c6b94d-s4q8p" Sep 08 11:32:44.773475 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:44.773445 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:32:49.765009 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:49.764974 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65cddccf-s4q77"] Sep 08 11:32:52.460618 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:32:52.460576 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-66f5s" podUID="645f6f42-ec34-4b66-bf8e-183601ddc1af" Sep 08 11:32:53.330738 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:53.330705 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-66f5s" Sep 08 11:32:54.240548 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:54.240518 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:54.240919 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:54.240573 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:32:57.361942 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:57.361916 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:32:57.364197 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:57.364178 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/645f6f42-ec34-4b66-bf8e-183601ddc1af-metrics-tls\") pod \"dns-default-66f5s\" (UID: \"645f6f42-ec34-4b66-bf8e-183601ddc1af\") " pod="openshift-dns/dns-default-66f5s" Sep 08 11:32:57.534185 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:57.534154 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2v99l\"" Sep 08 11:32:57.542508 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:57.542490 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-66f5s" Sep 08 11:32:57.843602 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:57.843578 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-66f5s"] Sep 08 11:32:57.846107 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:32:57.846076 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645f6f42_ec34_4b66_bf8e_183601ddc1af.slice/crio-32f40c782d4fde047e50d36f3cf8b105978db3ac3dfcdd2b939d41993200302b WatchSource:0}: Error finding container 32f40c782d4fde047e50d36f3cf8b105978db3ac3dfcdd2b939d41993200302b: Status 404 returned error can't find the container with id 32f40c782d4fde047e50d36f3cf8b105978db3ac3dfcdd2b939d41993200302b Sep 08 11:32:58.350873 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:32:58.350841 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-66f5s" event={"ID":"645f6f42-ec34-4b66-bf8e-183601ddc1af","Type":"ContainerStarted","Data":"32f40c782d4fde047e50d36f3cf8b105978db3ac3dfcdd2b939d41993200302b"} Sep 08 11:33:00.360674 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:00.360633 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-66f5s" event={"ID":"645f6f42-ec34-4b66-bf8e-183601ddc1af","Type":"ContainerStarted","Data":"f6aca2cfa61ae1cba8849076a14b022826ba4b43c15ef18493de0ca06aba50d1"} Sep 08 11:33:00.361039 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:00.360697 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-66f5s" event={"ID":"645f6f42-ec34-4b66-bf8e-183601ddc1af","Type":"ContainerStarted","Data":"15d35a96e32f9aa308dbf227d55bf6aae50dd621a570d796ddd3f2c6dcf91a1b"} Sep 08 11:33:00.361039 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:00.360724 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-66f5s" Sep 08 11:33:00.382914 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:00.382863 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-66f5s" podStartSLOduration=129.844769342 podStartE2EDuration="2m11.382848792s" totalImagesPullingTime="1.53807945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:33:00.381575218 +0000 UTC m=+169.240762562" watchObservedRunningTime="2026-09-08 11:33:00.382848792 +0000 UTC m=+169.242036109" Sep 08 11:33:05.296706 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.296621 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bf4f4cdb7-l6r9s" podUID="b535cef7-8dba-4819-b9a9-90d1c6319c19" containerName="console" containerID="cri-o://22208e87ef9603122afdf4ea438ea6e56ee09320eb016d1177e81ae6ec93f85a" gracePeriod=15 Sep 08 11:33:05.559348 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.559288 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:33:05.627469 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.627435 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-config\") pod \"b535cef7-8dba-4819-b9a9-90d1c6319c19\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " Sep 08 11:33:05.627643 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.627480 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-oauth-serving-cert\") pod \"b535cef7-8dba-4819-b9a9-90d1c6319c19\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " Sep 08 11:33:05.627643 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.627530 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcn4q\" (UniqueName: \"kubernetes.io/projected/b535cef7-8dba-4819-b9a9-90d1c6319c19-kube-api-access-gcn4q\") pod \"b535cef7-8dba-4819-b9a9-90d1c6319c19\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " Sep 08 11:33:05.627643 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.627551 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-serving-cert\") pod \"b535cef7-8dba-4819-b9a9-90d1c6319c19\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " Sep 08 11:33:05.627643 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.627590 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-oauth-config\") pod \"b535cef7-8dba-4819-b9a9-90d1c6319c19\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " Sep 08 11:33:05.627643 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.627620 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-trusted-ca-bundle\") pod \"b535cef7-8dba-4819-b9a9-90d1c6319c19\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " Sep 08 11:33:05.627882 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.627759 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-service-ca\") pod \"b535cef7-8dba-4819-b9a9-90d1c6319c19\" (UID: \"b535cef7-8dba-4819-b9a9-90d1c6319c19\") " Sep 08 11:33:05.627943 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.627876 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-config" (OuterVolumeSpecName: "console-config") pod "b535cef7-8dba-4819-b9a9-90d1c6319c19" (UID: "b535cef7-8dba-4819-b9a9-90d1c6319c19"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:33:05.628002 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.627916 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b535cef7-8dba-4819-b9a9-90d1c6319c19" (UID: "b535cef7-8dba-4819-b9a9-90d1c6319c19"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:33:05.628162 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.628053 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:33:05.628257 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.628173 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-oauth-serving-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:33:05.628257 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.628127 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b535cef7-8dba-4819-b9a9-90d1c6319c19" (UID: "b535cef7-8dba-4819-b9a9-90d1c6319c19"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:33:05.628257 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.628212 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-service-ca" (OuterVolumeSpecName: "service-ca") pod "b535cef7-8dba-4819-b9a9-90d1c6319c19" (UID: "b535cef7-8dba-4819-b9a9-90d1c6319c19"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:33:05.629953 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.629921 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b535cef7-8dba-4819-b9a9-90d1c6319c19" (UID: "b535cef7-8dba-4819-b9a9-90d1c6319c19"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:33:05.630048 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.629985 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b535cef7-8dba-4819-b9a9-90d1c6319c19-kube-api-access-gcn4q" (OuterVolumeSpecName: "kube-api-access-gcn4q") pod "b535cef7-8dba-4819-b9a9-90d1c6319c19" (UID: "b535cef7-8dba-4819-b9a9-90d1c6319c19"). InnerVolumeSpecName "kube-api-access-gcn4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:33:05.630048 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.629997 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b535cef7-8dba-4819-b9a9-90d1c6319c19" (UID: "b535cef7-8dba-4819-b9a9-90d1c6319c19"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:33:05.728583 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.728547 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-serving-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:33:05.728583 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.728576 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b535cef7-8dba-4819-b9a9-90d1c6319c19-console-oauth-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:33:05.728583 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.728585 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-trusted-ca-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:33:05.728802 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.728594 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b535cef7-8dba-4819-b9a9-90d1c6319c19-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:33:05.728802 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:05.728604 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcn4q\" (UniqueName: \"kubernetes.io/projected/b535cef7-8dba-4819-b9a9-90d1c6319c19-kube-api-access-gcn4q\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:33:06.384074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:06.384033 2564 generic.go:356] "Generic (PLEG): container finished" podID="b535cef7-8dba-4819-b9a9-90d1c6319c19" containerID="22208e87ef9603122afdf4ea438ea6e56ee09320eb016d1177e81ae6ec93f85a" exitCode=0 Sep 08 11:33:06.384560 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:06.384076 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf4f4cdb7-l6r9s" event={"ID":"b535cef7-8dba-4819-b9a9-90d1c6319c19","Type":"ContainerDied","Data":"22208e87ef9603122afdf4ea438ea6e56ee09320eb016d1177e81ae6ec93f85a"} Sep 08 11:33:06.384560 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:06.384099 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf4f4cdb7-l6r9s" Sep 08 11:33:06.384560 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:06.384131 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf4f4cdb7-l6r9s" event={"ID":"b535cef7-8dba-4819-b9a9-90d1c6319c19","Type":"ContainerDied","Data":"b62b5568e68ee40de0d05cf30e5595b9eb62d497cbfc2ba46148182fd14350fb"} Sep 08 11:33:06.384560 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:06.384155 2564 scope.go:117] "RemoveContainer" containerID="22208e87ef9603122afdf4ea438ea6e56ee09320eb016d1177e81ae6ec93f85a" Sep 08 11:33:06.391877 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:06.391853 2564 scope.go:117] "RemoveContainer" containerID="22208e87ef9603122afdf4ea438ea6e56ee09320eb016d1177e81ae6ec93f85a" Sep 08 11:33:06.392155 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:33:06.392135 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22208e87ef9603122afdf4ea438ea6e56ee09320eb016d1177e81ae6ec93f85a\": container with ID starting with 22208e87ef9603122afdf4ea438ea6e56ee09320eb016d1177e81ae6ec93f85a not found: ID does not exist" containerID="22208e87ef9603122afdf4ea438ea6e56ee09320eb016d1177e81ae6ec93f85a" Sep 08 11:33:06.392219 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:06.392163 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22208e87ef9603122afdf4ea438ea6e56ee09320eb016d1177e81ae6ec93f85a"} err="failed to get container status \"22208e87ef9603122afdf4ea438ea6e56ee09320eb016d1177e81ae6ec93f85a\": rpc error: code = NotFound desc = could not find container \"22208e87ef9603122afdf4ea438ea6e56ee09320eb016d1177e81ae6ec93f85a\": container with ID starting with 22208e87ef9603122afdf4ea438ea6e56ee09320eb016d1177e81ae6ec93f85a not found: ID does not exist" Sep 08 11:33:06.406692 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:06.406671 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf4f4cdb7-l6r9s"] Sep 08 11:33:06.412439 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:06.412419 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bf4f4cdb7-l6r9s"] Sep 08 11:33:07.651363 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:07.651332 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b535cef7-8dba-4819-b9a9-90d1c6319c19" path="/var/lib/kubelet/pods/b535cef7-8dba-4819-b9a9-90d1c6319c19/volumes" Sep 08 11:33:10.367103 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:10.367068 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-66f5s" Sep 08 11:33:14.245819 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:14.245782 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:33:14.249580 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:14.249558 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6dd97df8-b75x6" Sep 08 11:33:14.785830 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:14.785775 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65cddccf-s4q77" podUID="1f6978d2-b089-4dfa-bfb8-936174c3bfb6" containerName="console" containerID="cri-o://ee33c75e96f48333c4f1fcfba9c3de822a97673686ca5d56080c19df867a561a" gracePeriod=15 Sep 08 11:33:15.078457 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.078427 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:33:15.212012 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.211972 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-oauth-config\") pod \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " Sep 08 11:33:15.212012 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.212019 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-config\") pod \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " Sep 08 11:33:15.212230 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.212039 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-oauth-serving-cert\") pod \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " Sep 08 11:33:15.212230 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.212057 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-service-ca\") pod \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " Sep 08 11:33:15.212230 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.212096 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-serving-cert\") pod \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " Sep 08 11:33:15.212230 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.212136 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jr2xl\" (UniqueName: \"kubernetes.io/projected/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-kube-api-access-jr2xl\") pod \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\" (UID: \"1f6978d2-b089-4dfa-bfb8-936174c3bfb6\") " Sep 08 11:33:15.212489 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.212458 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-service-ca" (OuterVolumeSpecName: "service-ca") pod "1f6978d2-b089-4dfa-bfb8-936174c3bfb6" (UID: "1f6978d2-b089-4dfa-bfb8-936174c3bfb6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:33:15.212598 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.212567 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1f6978d2-b089-4dfa-bfb8-936174c3bfb6" (UID: "1f6978d2-b089-4dfa-bfb8-936174c3bfb6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:33:15.212598 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.212581 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-config" (OuterVolumeSpecName: "console-config") pod "1f6978d2-b089-4dfa-bfb8-936174c3bfb6" (UID: "1f6978d2-b089-4dfa-bfb8-936174c3bfb6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:33:15.214181 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.214155 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1f6978d2-b089-4dfa-bfb8-936174c3bfb6" (UID: "1f6978d2-b089-4dfa-bfb8-936174c3bfb6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:33:15.214328 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.214313 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-kube-api-access-jr2xl" (OuterVolumeSpecName: "kube-api-access-jr2xl") pod "1f6978d2-b089-4dfa-bfb8-936174c3bfb6" (UID: "1f6978d2-b089-4dfa-bfb8-936174c3bfb6"). InnerVolumeSpecName "kube-api-access-jr2xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:33:15.214403 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.214350 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1f6978d2-b089-4dfa-bfb8-936174c3bfb6" (UID: "1f6978d2-b089-4dfa-bfb8-936174c3bfb6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:33:15.313615 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.313535 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-oauth-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:33:15.313615 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.313564 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:33:15.313615 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.313573 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-oauth-serving-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:33:15.313615 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.313582 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:33:15.313615 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.313590 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-console-serving-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:33:15.313615 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.313598 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr2xl\" (UniqueName: \"kubernetes.io/projected/1f6978d2-b089-4dfa-bfb8-936174c3bfb6-kube-api-access-jr2xl\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:33:15.422140 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.422102 2564 generic.go:356] "Generic (PLEG): container finished" podID="1f6978d2-b089-4dfa-bfb8-936174c3bfb6" containerID="ee33c75e96f48333c4f1fcfba9c3de822a97673686ca5d56080c19df867a561a" exitCode=0 Sep 08 11:33:15.422140 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.422145 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cddccf-s4q77" Sep 08 11:33:15.422328 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.422184 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cddccf-s4q77" event={"ID":"1f6978d2-b089-4dfa-bfb8-936174c3bfb6","Type":"ContainerDied","Data":"ee33c75e96f48333c4f1fcfba9c3de822a97673686ca5d56080c19df867a561a"} Sep 08 11:33:15.422328 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.422233 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cddccf-s4q77" event={"ID":"1f6978d2-b089-4dfa-bfb8-936174c3bfb6","Type":"ContainerDied","Data":"ca40731f100c7d27593aec039c2d66b61f8996b61b47ef2eccc3caff5c6a7eb9"} Sep 08 11:33:15.422328 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.422253 2564 scope.go:117] "RemoveContainer" containerID="ee33c75e96f48333c4f1fcfba9c3de822a97673686ca5d56080c19df867a561a" Sep 08 11:33:15.431905 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.431889 2564 scope.go:117] "RemoveContainer" containerID="ee33c75e96f48333c4f1fcfba9c3de822a97673686ca5d56080c19df867a561a" Sep 08 11:33:15.432242 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:33:15.432218 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee33c75e96f48333c4f1fcfba9c3de822a97673686ca5d56080c19df867a561a\": container with ID starting with ee33c75e96f48333c4f1fcfba9c3de822a97673686ca5d56080c19df867a561a not found: ID does not exist" containerID="ee33c75e96f48333c4f1fcfba9c3de822a97673686ca5d56080c19df867a561a" Sep 08 11:33:15.432313 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.432252 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee33c75e96f48333c4f1fcfba9c3de822a97673686ca5d56080c19df867a561a"} err="failed to get container status \"ee33c75e96f48333c4f1fcfba9c3de822a97673686ca5d56080c19df867a561a\": rpc error: code = NotFound desc = could not find container \"ee33c75e96f48333c4f1fcfba9c3de822a97673686ca5d56080c19df867a561a\": container with ID starting with ee33c75e96f48333c4f1fcfba9c3de822a97673686ca5d56080c19df867a561a not found: ID does not exist" Sep 08 11:33:15.447168 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.447142 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65cddccf-s4q77"] Sep 08 11:33:15.452123 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.452103 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65cddccf-s4q77"] Sep 08 11:33:15.651728 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:15.651696 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f6978d2-b089-4dfa-bfb8-936174c3bfb6" path="/var/lib/kubelet/pods/1f6978d2-b089-4dfa-bfb8-936174c3bfb6/volumes" Sep 08 11:33:18.435335 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:18.435304 2564 generic.go:356] "Generic (PLEG): container finished" podID="de336abd-d8f8-429f-ab55-0041367141c5" containerID="626d5340505db1c7b5ee258b035f2806037c78fa4b001e167b53d47b1ce87392" exitCode=0 Sep 08 11:33:18.435693 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:18.435390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" event={"ID":"de336abd-d8f8-429f-ab55-0041367141c5","Type":"ContainerDied","Data":"626d5340505db1c7b5ee258b035f2806037c78fa4b001e167b53d47b1ce87392"} Sep 08 11:33:18.435693 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:18.435680 2564 scope.go:117] "RemoveContainer" containerID="626d5340505db1c7b5ee258b035f2806037c78fa4b001e167b53d47b1ce87392" Sep 08 11:33:19.440805 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:19.440763 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-zspv5" event={"ID":"de336abd-d8f8-429f-ab55-0041367141c5","Type":"ContainerStarted","Data":"ac03233d4b70d4a9896620f732519f579526317b1307483eca51d0c7c1997671"} Sep 08 11:33:19.442261 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:19.442234 2564 generic.go:356] "Generic (PLEG): container finished" podID="f44bd9c6-f403-43f1-820f-e49f9ded15e6" containerID="e4a14522358969b367b02a7857754e14fe87c8223f3c2d94a3f84827ee4a6397" exitCode=0 Sep 08 11:33:19.442407 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:19.442311 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" event={"ID":"f44bd9c6-f403-43f1-820f-e49f9ded15e6","Type":"ContainerDied","Data":"e4a14522358969b367b02a7857754e14fe87c8223f3c2d94a3f84827ee4a6397"} Sep 08 11:33:19.442665 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:19.442649 2564 scope.go:117] "RemoveContainer" containerID="e4a14522358969b367b02a7857754e14fe87c8223f3c2d94a3f84827ee4a6397" Sep 08 11:33:20.447691 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:33:20.447657 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-p5jkx" event={"ID":"f44bd9c6-f403-43f1-820f-e49f9ded15e6","Type":"ContainerStarted","Data":"2ab6bb7988d6e33c4ad24c70a4cea1ff9c3aafed55e5071707404bae0e3fa090"} Sep 08 11:34:08.067538 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.067487 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-756f55cbf4-kqcs8"] Sep 08 11:34:08.068157 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.067784 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="660f1576-9451-4015-b19c-640193d50ccc" containerName="registry" Sep 08 11:34:08.068157 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.067800 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="660f1576-9451-4015-b19c-640193d50ccc" containerName="registry" Sep 08 11:34:08.068157 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.067818 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f6978d2-b089-4dfa-bfb8-936174c3bfb6" containerName="console" Sep 08 11:34:08.068157 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.067826 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f6978d2-b089-4dfa-bfb8-936174c3bfb6" containerName="console" Sep 08 11:34:08.068157 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.067857 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b535cef7-8dba-4819-b9a9-90d1c6319c19" containerName="console" Sep 08 11:34:08.068157 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.067866 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b535cef7-8dba-4819-b9a9-90d1c6319c19" containerName="console" Sep 08 11:34:08.068157 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.067938 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b535cef7-8dba-4819-b9a9-90d1c6319c19" containerName="console" Sep 08 11:34:08.068157 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.067947 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="660f1576-9451-4015-b19c-640193d50ccc" containerName="registry" Sep 08 11:34:08.068157 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.067955 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f6978d2-b089-4dfa-bfb8-936174c3bfb6" containerName="console" Sep 08 11:34:08.069674 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.069656 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.072466 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.072444 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 11:34:08.072627 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.072602 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 11:34:08.073651 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.073633 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 11:34:08.073815 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.073791 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 11:34:08.073959 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.073938 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 11:34:08.074029 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.073942 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4fm94\"" Sep 08 11:34:08.080764 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.080646 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 08 11:34:08.085703 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.085678 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-756f55cbf4-kqcs8"] Sep 08 11:34:08.107197 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.107161 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ckrn\" (UniqueName: \"kubernetes.io/projected/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-kube-api-access-2ckrn\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.107197 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.107200 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-oauth-config\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.107422 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.107231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-service-ca\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.107422 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.107291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-trusted-ca-bundle\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.107422 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.107325 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-config\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.107422 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.107354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-serving-cert\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.107422 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.107395 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-oauth-serving-cert\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.207955 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.207921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ckrn\" (UniqueName: \"kubernetes.io/projected/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-kube-api-access-2ckrn\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.207955 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.207966 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-oauth-config\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.208229 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.208009 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-service-ca\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.208229 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.208028 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-trusted-ca-bundle\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.208229 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.208052 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-config\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.208229 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.208083 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-serving-cert\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.208229 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.208112 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-oauth-serving-cert\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.208908 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.208881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-service-ca\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.209031 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.208881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-config\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.209148 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.209128 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-oauth-serving-cert\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.209190 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.209164 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-trusted-ca-bundle\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.210785 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.210759 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-serving-cert\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.211329 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.211307 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-oauth-config\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.217283 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.217258 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ckrn\" (UniqueName: \"kubernetes.io/projected/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-kube-api-access-2ckrn\") pod \"console-756f55cbf4-kqcs8\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.380884 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.380789 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:08.655213 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:08.655086 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-756f55cbf4-kqcs8"] Sep 08 11:34:08.657425 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:34:08.657401 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d67e861_b4e2_4ac8_84e4_4ec9ca82a93b.slice/crio-eacc463bef8c77df2a9c7632b3dfe4972606ae5ce9f1c6931857da194364f27a WatchSource:0}: Error finding container eacc463bef8c77df2a9c7632b3dfe4972606ae5ce9f1c6931857da194364f27a: Status 404 returned error can't find the container with id eacc463bef8c77df2a9c7632b3dfe4972606ae5ce9f1c6931857da194364f27a Sep 08 11:34:09.630562 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:09.630527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-756f55cbf4-kqcs8" event={"ID":"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b","Type":"ContainerStarted","Data":"d1a692baff5d9f525f1feb7c17b3d91362fa31fee07640979f0ed594b7c8302e"} Sep 08 11:34:09.630941 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:09.630576 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-756f55cbf4-kqcs8" event={"ID":"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b","Type":"ContainerStarted","Data":"eacc463bef8c77df2a9c7632b3dfe4972606ae5ce9f1c6931857da194364f27a"} Sep 08 11:34:09.667674 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:09.667617 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-756f55cbf4-kqcs8" podStartSLOduration=1.667598891 podStartE2EDuration="1.667598891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:34:09.667454147 +0000 UTC m=+238.526641486" watchObservedRunningTime="2026-09-08 11:34:09.667598891 +0000 UTC m=+238.526786210" Sep 08 11:34:18.381872 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:18.381830 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:18.382421 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:18.381917 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:18.386883 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:18.386860 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:34:18.666046 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:34:18.666018 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:35:01.114930 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.114890 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx"] Sep 08 11:35:01.117385 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.117351 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" Sep 08 11:35:01.120167 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.120137 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sjmgb\"" Sep 08 11:35:01.120167 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.120161 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 11:35:01.121238 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.121215 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 11:35:01.127363 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.127341 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx"] Sep 08 11:35:01.222894 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.222859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b712499d-220a-4d8e-ad81-e916da125629-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx\" (UID: \"b712499d-220a-4d8e-ad81-e916da125629\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" Sep 08 11:35:01.223070 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.222906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b712499d-220a-4d8e-ad81-e916da125629-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx\" (UID: \"b712499d-220a-4d8e-ad81-e916da125629\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" Sep 08 11:35:01.223070 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.222925 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54p6f\" (UniqueName: \"kubernetes.io/projected/b712499d-220a-4d8e-ad81-e916da125629-kube-api-access-54p6f\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx\" (UID: \"b712499d-220a-4d8e-ad81-e916da125629\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" Sep 08 11:35:01.324256 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.324215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b712499d-220a-4d8e-ad81-e916da125629-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx\" (UID: \"b712499d-220a-4d8e-ad81-e916da125629\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" Sep 08 11:35:01.324430 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.324282 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b712499d-220a-4d8e-ad81-e916da125629-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx\" (UID: \"b712499d-220a-4d8e-ad81-e916da125629\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" Sep 08 11:35:01.324430 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.324350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54p6f\" (UniqueName: \"kubernetes.io/projected/b712499d-220a-4d8e-ad81-e916da125629-kube-api-access-54p6f\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx\" (UID: \"b712499d-220a-4d8e-ad81-e916da125629\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" Sep 08 11:35:01.324628 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.324611 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b712499d-220a-4d8e-ad81-e916da125629-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx\" (UID: \"b712499d-220a-4d8e-ad81-e916da125629\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" Sep 08 11:35:01.324667 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.324642 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b712499d-220a-4d8e-ad81-e916da125629-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx\" (UID: \"b712499d-220a-4d8e-ad81-e916da125629\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" Sep 08 11:35:01.333858 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.333832 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54p6f\" (UniqueName: \"kubernetes.io/projected/b712499d-220a-4d8e-ad81-e916da125629-kube-api-access-54p6f\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx\" (UID: \"b712499d-220a-4d8e-ad81-e916da125629\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" Sep 08 11:35:01.426107 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.426055 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" Sep 08 11:35:01.700528 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.700503 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx"] Sep 08 11:35:01.703168 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:35:01.703136 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb712499d_220a_4d8e_ad81_e916da125629.slice/crio-914793f910ee1e0238a90342b905d3f836c8d87515cd30894f9ec91f04222bc4 WatchSource:0}: Error finding container 914793f910ee1e0238a90342b905d3f836c8d87515cd30894f9ec91f04222bc4: Status 404 returned error can't find the container with id 914793f910ee1e0238a90342b905d3f836c8d87515cd30894f9ec91f04222bc4 Sep 08 11:35:01.824846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:01.824812 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" event={"ID":"b712499d-220a-4d8e-ad81-e916da125629","Type":"ContainerStarted","Data":"914793f910ee1e0238a90342b905d3f836c8d87515cd30894f9ec91f04222bc4"} Sep 08 11:35:07.851168 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:07.851070 2564 generic.go:356] "Generic (PLEG): container finished" podID="b712499d-220a-4d8e-ad81-e916da125629" containerID="44c4a7b4a6b6f5d3b0577e8221deef676adaa8780cc6503cc5aec798de5a62d9" exitCode=0 Sep 08 11:35:07.851649 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:07.851161 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" event={"ID":"b712499d-220a-4d8e-ad81-e916da125629","Type":"ContainerDied","Data":"44c4a7b4a6b6f5d3b0577e8221deef676adaa8780cc6503cc5aec798de5a62d9"} Sep 08 11:35:10.864825 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:10.864788 2564 generic.go:356] "Generic (PLEG): container finished" podID="b712499d-220a-4d8e-ad81-e916da125629" containerID="bd9607ad05605070fea379d076270b922bbb5c149aa3ae287c1f3f4a74332372" exitCode=0 Sep 08 11:35:10.865258 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:10.864880 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" event={"ID":"b712499d-220a-4d8e-ad81-e916da125629","Type":"ContainerDied","Data":"bd9607ad05605070fea379d076270b922bbb5c149aa3ae287c1f3f4a74332372"} Sep 08 11:35:11.524067 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:11.524034 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 11:35:11.527268 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:11.527241 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 11:35:11.536119 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:11.536095 2564 kubelet.go:1648] "Image garbage collection succeeded" Sep 08 11:35:17.895383 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:17.895336 2564 generic.go:356] "Generic (PLEG): container finished" podID="b712499d-220a-4d8e-ad81-e916da125629" containerID="201bffe83f6c73bfa9aec56a9595b597af861b4ec8868e72f794437d97bdc7d9" exitCode=0 Sep 08 11:35:17.895781 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:17.895402 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" event={"ID":"b712499d-220a-4d8e-ad81-e916da125629","Type":"ContainerDied","Data":"201bffe83f6c73bfa9aec56a9595b597af861b4ec8868e72f794437d97bdc7d9"} Sep 08 11:35:19.020515 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:19.020490 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" Sep 08 11:35:19.078848 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:19.078822 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54p6f\" (UniqueName: \"kubernetes.io/projected/b712499d-220a-4d8e-ad81-e916da125629-kube-api-access-54p6f\") pod \"b712499d-220a-4d8e-ad81-e916da125629\" (UID: \"b712499d-220a-4d8e-ad81-e916da125629\") " Sep 08 11:35:19.078979 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:19.078889 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b712499d-220a-4d8e-ad81-e916da125629-bundle\") pod \"b712499d-220a-4d8e-ad81-e916da125629\" (UID: \"b712499d-220a-4d8e-ad81-e916da125629\") " Sep 08 11:35:19.078979 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:19.078938 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b712499d-220a-4d8e-ad81-e916da125629-util\") pod \"b712499d-220a-4d8e-ad81-e916da125629\" (UID: \"b712499d-220a-4d8e-ad81-e916da125629\") " Sep 08 11:35:19.079678 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:19.079647 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b712499d-220a-4d8e-ad81-e916da125629-bundle" (OuterVolumeSpecName: "bundle") pod "b712499d-220a-4d8e-ad81-e916da125629" (UID: "b712499d-220a-4d8e-ad81-e916da125629"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:35:19.081073 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:19.081051 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b712499d-220a-4d8e-ad81-e916da125629-kube-api-access-54p6f" (OuterVolumeSpecName: "kube-api-access-54p6f") pod "b712499d-220a-4d8e-ad81-e916da125629" (UID: "b712499d-220a-4d8e-ad81-e916da125629"). InnerVolumeSpecName "kube-api-access-54p6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:35:19.083171 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:19.083147 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b712499d-220a-4d8e-ad81-e916da125629-util" (OuterVolumeSpecName: "util") pod "b712499d-220a-4d8e-ad81-e916da125629" (UID: "b712499d-220a-4d8e-ad81-e916da125629"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:35:19.179625 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:19.179557 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b712499d-220a-4d8e-ad81-e916da125629-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:35:19.179625 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:19.179585 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b712499d-220a-4d8e-ad81-e916da125629-util\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:35:19.179625 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:19.179594 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54p6f\" (UniqueName: \"kubernetes.io/projected/b712499d-220a-4d8e-ad81-e916da125629-kube-api-access-54p6f\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:35:19.905175 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:19.905148 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" Sep 08 11:35:19.905329 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:19.905146 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540h2fbx" event={"ID":"b712499d-220a-4d8e-ad81-e916da125629","Type":"ContainerDied","Data":"914793f910ee1e0238a90342b905d3f836c8d87515cd30894f9ec91f04222bc4"} Sep 08 11:35:19.905329 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:19.905262 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="914793f910ee1e0238a90342b905d3f836c8d87515cd30894f9ec91f04222bc4" Sep 08 11:35:24.102720 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.102687 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfn78"] Sep 08 11:35:24.103109 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.102954 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b712499d-220a-4d8e-ad81-e916da125629" containerName="pull" Sep 08 11:35:24.103109 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.102972 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b712499d-220a-4d8e-ad81-e916da125629" containerName="pull" Sep 08 11:35:24.103109 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.102987 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b712499d-220a-4d8e-ad81-e916da125629" containerName="util" Sep 08 11:35:24.103109 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.102992 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b712499d-220a-4d8e-ad81-e916da125629" containerName="util" Sep 08 11:35:24.103109 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.103002 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b712499d-220a-4d8e-ad81-e916da125629" containerName="extract" Sep 08 11:35:24.103109 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.103007 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b712499d-220a-4d8e-ad81-e916da125629" containerName="extract" Sep 08 11:35:24.103109 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.103059 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b712499d-220a-4d8e-ad81-e916da125629" containerName="extract" Sep 08 11:35:24.104782 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.104766 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfn78" Sep 08 11:35:24.107509 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.107487 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 08 11:35:24.107638 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.107566 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 08 11:35:24.107638 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.107571 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9t24r\"" Sep 08 11:35:24.119235 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.119211 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfn78"] Sep 08 11:35:24.216175 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.216145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b04a20f-3bca-420f-b850-1a7b9adcee29-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bfn78\" (UID: \"9b04a20f-3bca-420f-b850-1a7b9adcee29\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfn78" Sep 08 11:35:24.216322 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.216189 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmp4t\" (UniqueName: \"kubernetes.io/projected/9b04a20f-3bca-420f-b850-1a7b9adcee29-kube-api-access-gmp4t\") pod \"cert-manager-operator-controller-manager-55c444bf86-bfn78\" (UID: \"9b04a20f-3bca-420f-b850-1a7b9adcee29\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfn78" Sep 08 11:35:24.316645 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.316609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmp4t\" (UniqueName: \"kubernetes.io/projected/9b04a20f-3bca-420f-b850-1a7b9adcee29-kube-api-access-gmp4t\") pod \"cert-manager-operator-controller-manager-55c444bf86-bfn78\" (UID: \"9b04a20f-3bca-420f-b850-1a7b9adcee29\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfn78" Sep 08 11:35:24.316826 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.316714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b04a20f-3bca-420f-b850-1a7b9adcee29-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bfn78\" (UID: \"9b04a20f-3bca-420f-b850-1a7b9adcee29\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfn78" Sep 08 11:35:24.317091 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.317070 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b04a20f-3bca-420f-b850-1a7b9adcee29-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bfn78\" (UID: \"9b04a20f-3bca-420f-b850-1a7b9adcee29\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfn78" Sep 08 11:35:24.325859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.325833 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmp4t\" (UniqueName: \"kubernetes.io/projected/9b04a20f-3bca-420f-b850-1a7b9adcee29-kube-api-access-gmp4t\") pod \"cert-manager-operator-controller-manager-55c444bf86-bfn78\" (UID: \"9b04a20f-3bca-420f-b850-1a7b9adcee29\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfn78" Sep 08 11:35:24.414215 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.414187 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfn78" Sep 08 11:35:24.691199 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.691125 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfn78"] Sep 08 11:35:24.694439 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:35:24.694407 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b04a20f_3bca_420f_b850_1a7b9adcee29.slice/crio-b8bfaade3a7b3c8d02e0b9252baee4a1090605dff3a464d56e6025e877c40d82 WatchSource:0}: Error finding container b8bfaade3a7b3c8d02e0b9252baee4a1090605dff3a464d56e6025e877c40d82: Status 404 returned error can't find the container with id b8bfaade3a7b3c8d02e0b9252baee4a1090605dff3a464d56e6025e877c40d82 Sep 08 11:35:24.697005 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.696988 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 11:35:24.925736 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:24.925699 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfn78" event={"ID":"9b04a20f-3bca-420f-b850-1a7b9adcee29","Type":"ContainerStarted","Data":"b8bfaade3a7b3c8d02e0b9252baee4a1090605dff3a464d56e6025e877c40d82"} Sep 08 11:35:26.936817 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:26.936778 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfn78" event={"ID":"9b04a20f-3bca-420f-b850-1a7b9adcee29","Type":"ContainerStarted","Data":"f8590f4e680e812647eabefdcb52fa383c0ea62d37984de6cf89161979826f13"} Sep 08 11:35:26.959817 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:26.959764 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bfn78" podStartSLOduration=1.40012007 podStartE2EDuration="2.959745376s" totalImagesPullingTime="1.559625306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:35:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:35:26.958988927 +0000 UTC m=+315.818176266" watchObservedRunningTime="2026-09-08 11:35:26.959745376 +0000 UTC m=+315.818932695" Sep 08 11:35:27.590785 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:27.590753 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-r86q9"] Sep 08 11:35:27.592910 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:27.592887 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-r86q9" Sep 08 11:35:27.595603 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:27.595578 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qg7xg\"" Sep 08 11:35:27.596718 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:27.596698 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 08 11:35:27.596794 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:27.596698 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 08 11:35:27.614005 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:27.613984 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-r86q9"] Sep 08 11:35:27.745026 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:27.744979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfcd8339-352b-469f-975e-10c25f8d9501-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-r86q9\" (UID: \"cfcd8339-352b-469f-975e-10c25f8d9501\") " pod="cert-manager/cert-manager-webhook-6486ffd997-r86q9" Sep 08 11:35:27.745168 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:27.745043 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg648\" (UniqueName: \"kubernetes.io/projected/cfcd8339-352b-469f-975e-10c25f8d9501-kube-api-access-mg648\") pod \"cert-manager-webhook-6486ffd997-r86q9\" (UID: \"cfcd8339-352b-469f-975e-10c25f8d9501\") " pod="cert-manager/cert-manager-webhook-6486ffd997-r86q9" Sep 08 11:35:27.845619 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:27.845547 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfcd8339-352b-469f-975e-10c25f8d9501-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-r86q9\" (UID: \"cfcd8339-352b-469f-975e-10c25f8d9501\") " pod="cert-manager/cert-manager-webhook-6486ffd997-r86q9" Sep 08 11:35:27.845619 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:27.845589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg648\" (UniqueName: \"kubernetes.io/projected/cfcd8339-352b-469f-975e-10c25f8d9501-kube-api-access-mg648\") pod \"cert-manager-webhook-6486ffd997-r86q9\" (UID: \"cfcd8339-352b-469f-975e-10c25f8d9501\") " pod="cert-manager/cert-manager-webhook-6486ffd997-r86q9" Sep 08 11:35:27.855197 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:27.855166 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg648\" (UniqueName: \"kubernetes.io/projected/cfcd8339-352b-469f-975e-10c25f8d9501-kube-api-access-mg648\") pod \"cert-manager-webhook-6486ffd997-r86q9\" (UID: \"cfcd8339-352b-469f-975e-10c25f8d9501\") " pod="cert-manager/cert-manager-webhook-6486ffd997-r86q9" Sep 08 11:35:27.855303 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:27.855238 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfcd8339-352b-469f-975e-10c25f8d9501-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-r86q9\" (UID: \"cfcd8339-352b-469f-975e-10c25f8d9501\") " pod="cert-manager/cert-manager-webhook-6486ffd997-r86q9" Sep 08 11:35:27.902213 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:27.902188 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-r86q9" Sep 08 11:35:28.163430 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.163359 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn"] Sep 08 11:35:28.166510 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.166490 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" Sep 08 11:35:28.168980 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.168956 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 11:35:28.169133 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.169118 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sjmgb\"" Sep 08 11:35:28.170196 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.170179 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 11:35:28.175785 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.175761 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn"] Sep 08 11:35:28.179243 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.179161 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-r86q9"] Sep 08 11:35:28.181880 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:35:28.181859 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfcd8339_352b_469f_975e_10c25f8d9501.slice/crio-46f4580d5f7d9a5dc66899dfb70ba1ceea02bd2ce7d286678e476d7fc3a9cc51 WatchSource:0}: Error finding container 46f4580d5f7d9a5dc66899dfb70ba1ceea02bd2ce7d286678e476d7fc3a9cc51: Status 404 returned error can't find the container with id 46f4580d5f7d9a5dc66899dfb70ba1ceea02bd2ce7d286678e476d7fc3a9cc51 Sep 08 11:35:28.249143 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.249114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql8bf\" (UniqueName: \"kubernetes.io/projected/642cd785-7892-48d1-80ab-c51ed7d2d757-kube-api-access-ql8bf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn\" (UID: \"642cd785-7892-48d1-80ab-c51ed7d2d757\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" Sep 08 11:35:28.249254 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.249149 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/642cd785-7892-48d1-80ab-c51ed7d2d757-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn\" (UID: \"642cd785-7892-48d1-80ab-c51ed7d2d757\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" Sep 08 11:35:28.249254 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.249183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/642cd785-7892-48d1-80ab-c51ed7d2d757-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn\" (UID: \"642cd785-7892-48d1-80ab-c51ed7d2d757\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" Sep 08 11:35:28.350033 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.350009 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql8bf\" (UniqueName: \"kubernetes.io/projected/642cd785-7892-48d1-80ab-c51ed7d2d757-kube-api-access-ql8bf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn\" (UID: \"642cd785-7892-48d1-80ab-c51ed7d2d757\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" Sep 08 11:35:28.350139 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.350044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/642cd785-7892-48d1-80ab-c51ed7d2d757-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn\" (UID: \"642cd785-7892-48d1-80ab-c51ed7d2d757\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" Sep 08 11:35:28.350139 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.350081 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/642cd785-7892-48d1-80ab-c51ed7d2d757-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn\" (UID: \"642cd785-7892-48d1-80ab-c51ed7d2d757\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" Sep 08 11:35:28.350436 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.350419 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/642cd785-7892-48d1-80ab-c51ed7d2d757-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn\" (UID: \"642cd785-7892-48d1-80ab-c51ed7d2d757\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" Sep 08 11:35:28.350480 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.350448 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/642cd785-7892-48d1-80ab-c51ed7d2d757-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn\" (UID: \"642cd785-7892-48d1-80ab-c51ed7d2d757\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" Sep 08 11:35:28.361668 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.361644 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql8bf\" (UniqueName: \"kubernetes.io/projected/642cd785-7892-48d1-80ab-c51ed7d2d757-kube-api-access-ql8bf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn\" (UID: \"642cd785-7892-48d1-80ab-c51ed7d2d757\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" Sep 08 11:35:28.475878 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.475813 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" Sep 08 11:35:28.744123 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.744101 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn"] Sep 08 11:35:28.745979 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:35:28.745955 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642cd785_7892_48d1_80ab_c51ed7d2d757.slice/crio-f3c15c92622d0f5aaf545e022901eb46996a2ce09bec3074bde9d6613b97d2b8 WatchSource:0}: Error finding container f3c15c92622d0f5aaf545e022901eb46996a2ce09bec3074bde9d6613b97d2b8: Status 404 returned error can't find the container with id f3c15c92622d0f5aaf545e022901eb46996a2ce09bec3074bde9d6613b97d2b8 Sep 08 11:35:28.947436 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.947397 2564 generic.go:356] "Generic (PLEG): container finished" podID="642cd785-7892-48d1-80ab-c51ed7d2d757" containerID="bb999dd48fc44673b5cb8cc8ebc90193563e482c4658615c4db0e451744f5264" exitCode=0 Sep 08 11:35:28.947606 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.947495 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" event={"ID":"642cd785-7892-48d1-80ab-c51ed7d2d757","Type":"ContainerDied","Data":"bb999dd48fc44673b5cb8cc8ebc90193563e482c4658615c4db0e451744f5264"} Sep 08 11:35:28.947606 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.947574 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" event={"ID":"642cd785-7892-48d1-80ab-c51ed7d2d757","Type":"ContainerStarted","Data":"f3c15c92622d0f5aaf545e022901eb46996a2ce09bec3074bde9d6613b97d2b8"} Sep 08 11:35:28.949179 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:28.949141 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-r86q9" event={"ID":"cfcd8339-352b-469f-975e-10c25f8d9501","Type":"ContainerStarted","Data":"46f4580d5f7d9a5dc66899dfb70ba1ceea02bd2ce7d286678e476d7fc3a9cc51"} Sep 08 11:35:31.964647 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:31.964620 2564 generic.go:356] "Generic (PLEG): container finished" podID="642cd785-7892-48d1-80ab-c51ed7d2d757" containerID="26a83682a12dbcf9cb5bff2c276f983d80fbe672721a85603273e09527d4ef75" exitCode=0 Sep 08 11:35:31.964986 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:31.964688 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" event={"ID":"642cd785-7892-48d1-80ab-c51ed7d2d757","Type":"ContainerDied","Data":"26a83682a12dbcf9cb5bff2c276f983d80fbe672721a85603273e09527d4ef75"} Sep 08 11:35:31.966301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:31.966268 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-r86q9" event={"ID":"cfcd8339-352b-469f-975e-10c25f8d9501","Type":"ContainerStarted","Data":"72ff116b38d61ed79a492bfe7b9702878a9b3cf4ed70f81779e541532efa6d06"} Sep 08 11:35:31.966416 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:31.966405 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-r86q9" Sep 08 11:35:32.002477 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:32.002428 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-r86q9" podStartSLOduration=1.419052121 podStartE2EDuration="5.00241195s" totalImagesPullingTime="3.583359829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:35:32.002124978 +0000 UTC m=+320.861312319" watchObservedRunningTime="2026-09-08 11:35:32.00241195 +0000 UTC m=+320.861599268" Sep 08 11:35:32.972802 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:32.972763 2564 generic.go:356] "Generic (PLEG): container finished" podID="642cd785-7892-48d1-80ab-c51ed7d2d757" containerID="208de814c0b7a90223fe8e9b813defecfd039b0e5cb84ff84597468dba370279" exitCode=0 Sep 08 11:35:32.973258 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:32.972845 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" event={"ID":"642cd785-7892-48d1-80ab-c51ed7d2d757","Type":"ContainerDied","Data":"208de814c0b7a90223fe8e9b813defecfd039b0e5cb84ff84597468dba370279"} Sep 08 11:35:34.102095 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.102074 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" Sep 08 11:35:34.201917 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.201886 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql8bf\" (UniqueName: \"kubernetes.io/projected/642cd785-7892-48d1-80ab-c51ed7d2d757-kube-api-access-ql8bf\") pod \"642cd785-7892-48d1-80ab-c51ed7d2d757\" (UID: \"642cd785-7892-48d1-80ab-c51ed7d2d757\") " Sep 08 11:35:34.202092 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.201955 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/642cd785-7892-48d1-80ab-c51ed7d2d757-util\") pod \"642cd785-7892-48d1-80ab-c51ed7d2d757\" (UID: \"642cd785-7892-48d1-80ab-c51ed7d2d757\") " Sep 08 11:35:34.202092 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.201978 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/642cd785-7892-48d1-80ab-c51ed7d2d757-bundle\") pod \"642cd785-7892-48d1-80ab-c51ed7d2d757\" (UID: \"642cd785-7892-48d1-80ab-c51ed7d2d757\") " Sep 08 11:35:34.202359 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.202335 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642cd785-7892-48d1-80ab-c51ed7d2d757-bundle" (OuterVolumeSpecName: "bundle") pod "642cd785-7892-48d1-80ab-c51ed7d2d757" (UID: "642cd785-7892-48d1-80ab-c51ed7d2d757"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:35:34.204053 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.204031 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642cd785-7892-48d1-80ab-c51ed7d2d757-kube-api-access-ql8bf" (OuterVolumeSpecName: "kube-api-access-ql8bf") pod "642cd785-7892-48d1-80ab-c51ed7d2d757" (UID: "642cd785-7892-48d1-80ab-c51ed7d2d757"). InnerVolumeSpecName "kube-api-access-ql8bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:35:34.208785 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.208757 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/642cd785-7892-48d1-80ab-c51ed7d2d757-util" (OuterVolumeSpecName: "util") pod "642cd785-7892-48d1-80ab-c51ed7d2d757" (UID: "642cd785-7892-48d1-80ab-c51ed7d2d757"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:35:34.302676 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.302604 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql8bf\" (UniqueName: \"kubernetes.io/projected/642cd785-7892-48d1-80ab-c51ed7d2d757-kube-api-access-ql8bf\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:35:34.302676 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.302631 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/642cd785-7892-48d1-80ab-c51ed7d2d757-util\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:35:34.302676 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.302642 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/642cd785-7892-48d1-80ab-c51ed7d2d757-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:35:34.781380 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.781334 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tlwqz"] Sep 08 11:35:34.781642 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.781629 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="642cd785-7892-48d1-80ab-c51ed7d2d757" containerName="pull" Sep 08 11:35:34.781688 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.781643 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="642cd785-7892-48d1-80ab-c51ed7d2d757" containerName="pull" Sep 08 11:35:34.781688 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.781663 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="642cd785-7892-48d1-80ab-c51ed7d2d757" containerName="util" Sep 08 11:35:34.781688 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.781669 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="642cd785-7892-48d1-80ab-c51ed7d2d757" containerName="util" Sep 08 11:35:34.781688 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.781679 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="642cd785-7892-48d1-80ab-c51ed7d2d757" containerName="extract" Sep 08 11:35:34.781688 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.781687 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="642cd785-7892-48d1-80ab-c51ed7d2d757" containerName="extract" Sep 08 11:35:34.781859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.781738 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="642cd785-7892-48d1-80ab-c51ed7d2d757" containerName="extract" Sep 08 11:35:34.784456 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.784439 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tlwqz" Sep 08 11:35:34.788321 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.788299 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-x6rxj\"" Sep 08 11:35:34.794098 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.794077 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tlwqz"] Sep 08 11:35:34.906289 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.906256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h97n7\" (UniqueName: \"kubernetes.io/projected/d4a8d192-d083-406d-b8e8-f4d99c8bd02e-kube-api-access-h97n7\") pod \"cert-manager-cainjector-559cdd8c87-tlwqz\" (UID: \"d4a8d192-d083-406d-b8e8-f4d99c8bd02e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tlwqz" Sep 08 11:35:34.906428 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.906299 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4a8d192-d083-406d-b8e8-f4d99c8bd02e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tlwqz\" (UID: \"d4a8d192-d083-406d-b8e8-f4d99c8bd02e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tlwqz" Sep 08 11:35:34.983479 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.983449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" event={"ID":"642cd785-7892-48d1-80ab-c51ed7d2d757","Type":"ContainerDied","Data":"f3c15c92622d0f5aaf545e022901eb46996a2ce09bec3074bde9d6613b97d2b8"} Sep 08 11:35:34.983583 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.983483 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcq7jn" Sep 08 11:35:34.983583 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:34.983492 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3c15c92622d0f5aaf545e022901eb46996a2ce09bec3074bde9d6613b97d2b8" Sep 08 11:35:35.006804 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:35.006781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h97n7\" (UniqueName: \"kubernetes.io/projected/d4a8d192-d083-406d-b8e8-f4d99c8bd02e-kube-api-access-h97n7\") pod \"cert-manager-cainjector-559cdd8c87-tlwqz\" (UID: \"d4a8d192-d083-406d-b8e8-f4d99c8bd02e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tlwqz" Sep 08 11:35:35.006917 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:35.006819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4a8d192-d083-406d-b8e8-f4d99c8bd02e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tlwqz\" (UID: \"d4a8d192-d083-406d-b8e8-f4d99c8bd02e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tlwqz" Sep 08 11:35:35.018213 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:35.018188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4a8d192-d083-406d-b8e8-f4d99c8bd02e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tlwqz\" (UID: \"d4a8d192-d083-406d-b8e8-f4d99c8bd02e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tlwqz" Sep 08 11:35:35.018317 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:35.018219 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h97n7\" (UniqueName: \"kubernetes.io/projected/d4a8d192-d083-406d-b8e8-f4d99c8bd02e-kube-api-access-h97n7\") pod \"cert-manager-cainjector-559cdd8c87-tlwqz\" (UID: \"d4a8d192-d083-406d-b8e8-f4d99c8bd02e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tlwqz" Sep 08 11:35:35.094076 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:35.094018 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tlwqz" Sep 08 11:35:35.378498 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:35.378475 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tlwqz"] Sep 08 11:35:35.380157 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:35:35.380132 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4a8d192_d083_406d_b8e8_f4d99c8bd02e.slice/crio-c56b045b4fd570f75ecab434f850c1f0e424034065c8e085c71fe76fa077dc2e WatchSource:0}: Error finding container c56b045b4fd570f75ecab434f850c1f0e424034065c8e085c71fe76fa077dc2e: Status 404 returned error can't find the container with id c56b045b4fd570f75ecab434f850c1f0e424034065c8e085c71fe76fa077dc2e Sep 08 11:35:35.989417 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:35.989379 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tlwqz" event={"ID":"d4a8d192-d083-406d-b8e8-f4d99c8bd02e","Type":"ContainerStarted","Data":"5e11bd9e2f25b37c126486f4fb1ac6fb51da663268da2047a412bc3c768c797d"} Sep 08 11:35:35.989586 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:35.989426 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tlwqz" event={"ID":"d4a8d192-d083-406d-b8e8-f4d99c8bd02e","Type":"ContainerStarted","Data":"c56b045b4fd570f75ecab434f850c1f0e424034065c8e085c71fe76fa077dc2e"} Sep 08 11:35:36.013023 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:36.012972 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tlwqz" podStartSLOduration=2.012957391 podStartE2EDuration="2.012957391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:35:36.012124542 +0000 UTC m=+324.871311861" watchObservedRunningTime="2026-09-08 11:35:36.012957391 +0000 UTC m=+324.872144709" Sep 08 11:35:37.515520 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:37.515487 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zhdzr"] Sep 08 11:35:37.521124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:37.521104 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-zhdzr" Sep 08 11:35:37.524524 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:37.524505 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-t6b2g\"" Sep 08 11:35:37.543125 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:37.543100 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zhdzr"] Sep 08 11:35:37.627875 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:37.627844 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf9ld\" (UniqueName: \"kubernetes.io/projected/2dbba133-d4c1-4413-99fc-3e6ec45ec655-kube-api-access-cf9ld\") pod \"cert-manager-55b4d954d5-zhdzr\" (UID: \"2dbba133-d4c1-4413-99fc-3e6ec45ec655\") " pod="cert-manager/cert-manager-55b4d954d5-zhdzr" Sep 08 11:35:37.628007 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:37.627908 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2dbba133-d4c1-4413-99fc-3e6ec45ec655-bound-sa-token\") pod \"cert-manager-55b4d954d5-zhdzr\" (UID: \"2dbba133-d4c1-4413-99fc-3e6ec45ec655\") " pod="cert-manager/cert-manager-55b4d954d5-zhdzr" Sep 08 11:35:37.728424 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:37.728363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2dbba133-d4c1-4413-99fc-3e6ec45ec655-bound-sa-token\") pod \"cert-manager-55b4d954d5-zhdzr\" (UID: \"2dbba133-d4c1-4413-99fc-3e6ec45ec655\") " pod="cert-manager/cert-manager-55b4d954d5-zhdzr" Sep 08 11:35:37.728563 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:37.728486 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9ld\" (UniqueName: \"kubernetes.io/projected/2dbba133-d4c1-4413-99fc-3e6ec45ec655-kube-api-access-cf9ld\") pod \"cert-manager-55b4d954d5-zhdzr\" (UID: \"2dbba133-d4c1-4413-99fc-3e6ec45ec655\") " pod="cert-manager/cert-manager-55b4d954d5-zhdzr" Sep 08 11:35:37.739822 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:37.739794 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2dbba133-d4c1-4413-99fc-3e6ec45ec655-bound-sa-token\") pod \"cert-manager-55b4d954d5-zhdzr\" (UID: \"2dbba133-d4c1-4413-99fc-3e6ec45ec655\") " pod="cert-manager/cert-manager-55b4d954d5-zhdzr" Sep 08 11:35:37.740089 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:37.740069 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9ld\" (UniqueName: \"kubernetes.io/projected/2dbba133-d4c1-4413-99fc-3e6ec45ec655-kube-api-access-cf9ld\") pod \"cert-manager-55b4d954d5-zhdzr\" (UID: \"2dbba133-d4c1-4413-99fc-3e6ec45ec655\") " pod="cert-manager/cert-manager-55b4d954d5-zhdzr" Sep 08 11:35:37.830164 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:37.830087 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-zhdzr" Sep 08 11:35:37.974807 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:37.974777 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-r86q9" Sep 08 11:35:38.102282 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:38.102230 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zhdzr"] Sep 08 11:35:38.104320 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:35:38.104289 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dbba133_d4c1_4413_99fc_3e6ec45ec655.slice/crio-75eea7e440b48df2511d8e146a6e2dbe88146da797f7848299466597bb50f3d4 WatchSource:0}: Error finding container 75eea7e440b48df2511d8e146a6e2dbe88146da797f7848299466597bb50f3d4: Status 404 returned error can't find the container with id 75eea7e440b48df2511d8e146a6e2dbe88146da797f7848299466597bb50f3d4 Sep 08 11:35:39.005111 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:39.005075 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-zhdzr" event={"ID":"2dbba133-d4c1-4413-99fc-3e6ec45ec655","Type":"ContainerStarted","Data":"63518d112ce4657413bfded4fe4c3687248283a1c6792814d30aa8abc446ba74"} Sep 08 11:35:39.005111 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:39.005112 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-zhdzr" event={"ID":"2dbba133-d4c1-4413-99fc-3e6ec45ec655","Type":"ContainerStarted","Data":"75eea7e440b48df2511d8e146a6e2dbe88146da797f7848299466597bb50f3d4"} Sep 08 11:35:39.036697 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:39.036644 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-zhdzr" podStartSLOduration=2.036628926 podStartE2EDuration="2.036628926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:35:39.035759133 +0000 UTC m=+327.894946452" watchObservedRunningTime="2026-09-08 11:35:39.036628926 +0000 UTC m=+327.895816243" Sep 08 11:35:51.159591 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.159549 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8"] Sep 08 11:35:51.166260 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.166236 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" Sep 08 11:35:51.169248 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.169225 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 11:35:51.169248 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.169226 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 11:35:51.170433 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.170414 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sjmgb\"" Sep 08 11:35:51.172556 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.172536 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8"] Sep 08 11:35:51.238958 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.238928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25e83822-b10b-4f3f-b3ae-580844a1d995-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8\" (UID: \"25e83822-b10b-4f3f-b3ae-580844a1d995\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" Sep 08 11:35:51.239101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.238979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc6mz\" (UniqueName: \"kubernetes.io/projected/25e83822-b10b-4f3f-b3ae-580844a1d995-kube-api-access-jc6mz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8\" (UID: \"25e83822-b10b-4f3f-b3ae-580844a1d995\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" Sep 08 11:35:51.239101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.239047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25e83822-b10b-4f3f-b3ae-580844a1d995-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8\" (UID: \"25e83822-b10b-4f3f-b3ae-580844a1d995\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" Sep 08 11:35:51.339477 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.339443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25e83822-b10b-4f3f-b3ae-580844a1d995-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8\" (UID: \"25e83822-b10b-4f3f-b3ae-580844a1d995\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" Sep 08 11:35:51.339610 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.339505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25e83822-b10b-4f3f-b3ae-580844a1d995-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8\" (UID: \"25e83822-b10b-4f3f-b3ae-580844a1d995\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" Sep 08 11:35:51.339610 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.339537 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc6mz\" (UniqueName: \"kubernetes.io/projected/25e83822-b10b-4f3f-b3ae-580844a1d995-kube-api-access-jc6mz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8\" (UID: \"25e83822-b10b-4f3f-b3ae-580844a1d995\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" Sep 08 11:35:51.339831 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.339809 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25e83822-b10b-4f3f-b3ae-580844a1d995-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8\" (UID: \"25e83822-b10b-4f3f-b3ae-580844a1d995\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" Sep 08 11:35:51.339871 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.339840 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25e83822-b10b-4f3f-b3ae-580844a1d995-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8\" (UID: \"25e83822-b10b-4f3f-b3ae-580844a1d995\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" Sep 08 11:35:51.349637 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.349610 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc6mz\" (UniqueName: \"kubernetes.io/projected/25e83822-b10b-4f3f-b3ae-580844a1d995-kube-api-access-jc6mz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8\" (UID: \"25e83822-b10b-4f3f-b3ae-580844a1d995\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" Sep 08 11:35:51.477033 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.476961 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" Sep 08 11:35:51.748422 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:51.748401 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8"] Sep 08 11:35:51.750609 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:35:51.750581 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e83822_b10b_4f3f_b3ae_580844a1d995.slice/crio-135ea13c5bba958ce7766be39a72ea54cb0c473f14909ddd5ce703bc66f2000a WatchSource:0}: Error finding container 135ea13c5bba958ce7766be39a72ea54cb0c473f14909ddd5ce703bc66f2000a: Status 404 returned error can't find the container with id 135ea13c5bba958ce7766be39a72ea54cb0c473f14909ddd5ce703bc66f2000a Sep 08 11:35:52.061240 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:52.061157 2564 generic.go:356] "Generic (PLEG): container finished" podID="25e83822-b10b-4f3f-b3ae-580844a1d995" containerID="a72ff33945f990297c1aac9fde303a7e20850df61cb1fa5c8b9cda79f27e12c2" exitCode=0 Sep 08 11:35:52.061240 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:52.061209 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" event={"ID":"25e83822-b10b-4f3f-b3ae-580844a1d995","Type":"ContainerDied","Data":"a72ff33945f990297c1aac9fde303a7e20850df61cb1fa5c8b9cda79f27e12c2"} Sep 08 11:35:52.061451 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:52.061245 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" event={"ID":"25e83822-b10b-4f3f-b3ae-580844a1d995","Type":"ContainerStarted","Data":"135ea13c5bba958ce7766be39a72ea54cb0c473f14909ddd5ce703bc66f2000a"} Sep 08 11:35:53.067311 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:53.067228 2564 generic.go:356] "Generic (PLEG): container finished" podID="25e83822-b10b-4f3f-b3ae-580844a1d995" containerID="4f03d0f372ffb3be8ae7331c0ef7c88fbb6521a0d43a4cda8c0715fb24b54724" exitCode=0 Sep 08 11:35:53.067311 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:53.067281 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" event={"ID":"25e83822-b10b-4f3f-b3ae-580844a1d995","Type":"ContainerDied","Data":"4f03d0f372ffb3be8ae7331c0ef7c88fbb6521a0d43a4cda8c0715fb24b54724"} Sep 08 11:35:54.073791 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:54.073758 2564 generic.go:356] "Generic (PLEG): container finished" podID="25e83822-b10b-4f3f-b3ae-580844a1d995" containerID="bda083e5815bf0f9a73d8f248053850dcecb04fd3401ee37aba02fb57fde28c8" exitCode=0 Sep 08 11:35:54.074220 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:54.073801 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" event={"ID":"25e83822-b10b-4f3f-b3ae-580844a1d995","Type":"ContainerDied","Data":"bda083e5815bf0f9a73d8f248053850dcecb04fd3401ee37aba02fb57fde28c8"} Sep 08 11:35:55.198605 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:55.198581 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" Sep 08 11:35:55.268938 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:55.268912 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc6mz\" (UniqueName: \"kubernetes.io/projected/25e83822-b10b-4f3f-b3ae-580844a1d995-kube-api-access-jc6mz\") pod \"25e83822-b10b-4f3f-b3ae-580844a1d995\" (UID: \"25e83822-b10b-4f3f-b3ae-580844a1d995\") " Sep 08 11:35:55.269064 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:55.268979 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25e83822-b10b-4f3f-b3ae-580844a1d995-bundle\") pod \"25e83822-b10b-4f3f-b3ae-580844a1d995\" (UID: \"25e83822-b10b-4f3f-b3ae-580844a1d995\") " Sep 08 11:35:55.269064 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:55.269025 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25e83822-b10b-4f3f-b3ae-580844a1d995-util\") pod \"25e83822-b10b-4f3f-b3ae-580844a1d995\" (UID: \"25e83822-b10b-4f3f-b3ae-580844a1d995\") " Sep 08 11:35:55.269877 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:55.269838 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e83822-b10b-4f3f-b3ae-580844a1d995-bundle" (OuterVolumeSpecName: "bundle") pod "25e83822-b10b-4f3f-b3ae-580844a1d995" (UID: "25e83822-b10b-4f3f-b3ae-580844a1d995"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:35:55.270942 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:55.270920 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e83822-b10b-4f3f-b3ae-580844a1d995-kube-api-access-jc6mz" (OuterVolumeSpecName: "kube-api-access-jc6mz") pod "25e83822-b10b-4f3f-b3ae-580844a1d995" (UID: "25e83822-b10b-4f3f-b3ae-580844a1d995"). InnerVolumeSpecName "kube-api-access-jc6mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:35:55.275001 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:55.274981 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e83822-b10b-4f3f-b3ae-580844a1d995-util" (OuterVolumeSpecName: "util") pod "25e83822-b10b-4f3f-b3ae-580844a1d995" (UID: "25e83822-b10b-4f3f-b3ae-580844a1d995"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:35:55.370416 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:55.370341 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc6mz\" (UniqueName: \"kubernetes.io/projected/25e83822-b10b-4f3f-b3ae-580844a1d995-kube-api-access-jc6mz\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:35:55.370416 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:55.370364 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25e83822-b10b-4f3f-b3ae-580844a1d995-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:35:55.370416 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:55.370389 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25e83822-b10b-4f3f-b3ae-580844a1d995-util\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:35:56.086696 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:56.086617 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" Sep 08 11:35:56.086696 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:56.086650 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcq9d8" event={"ID":"25e83822-b10b-4f3f-b3ae-580844a1d995","Type":"ContainerDied","Data":"135ea13c5bba958ce7766be39a72ea54cb0c473f14909ddd5ce703bc66f2000a"} Sep 08 11:35:56.086696 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:35:56.086698 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="135ea13c5bba958ce7766be39a72ea54cb0c473f14909ddd5ce703bc66f2000a" Sep 08 11:36:00.408714 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.408680 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc"] Sep 08 11:36:00.409079 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.408973 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e83822-b10b-4f3f-b3ae-580844a1d995" containerName="extract" Sep 08 11:36:00.409079 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.408984 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e83822-b10b-4f3f-b3ae-580844a1d995" containerName="extract" Sep 08 11:36:00.409079 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.409003 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e83822-b10b-4f3f-b3ae-580844a1d995" containerName="pull" Sep 08 11:36:00.409079 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.409009 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e83822-b10b-4f3f-b3ae-580844a1d995" containerName="pull" Sep 08 11:36:00.409079 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.409021 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e83822-b10b-4f3f-b3ae-580844a1d995" containerName="util" Sep 08 11:36:00.409079 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.409026 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e83822-b10b-4f3f-b3ae-580844a1d995" containerName="util" Sep 08 11:36:00.409079 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.409074 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="25e83822-b10b-4f3f-b3ae-580844a1d995" containerName="extract" Sep 08 11:36:00.411103 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.411086 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" Sep 08 11:36:00.414294 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.414272 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sjmgb\"" Sep 08 11:36:00.414584 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.414567 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 11:36:00.415397 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.415382 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 11:36:00.425779 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.425756 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc"] Sep 08 11:36:00.513025 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.512988 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scszf\" (UniqueName: \"kubernetes.io/projected/5552ce0b-52e6-4818-aa19-ca2872fb825b-kube-api-access-scszf\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc\" (UID: \"5552ce0b-52e6-4818-aa19-ca2872fb825b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" Sep 08 11:36:00.513184 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.513028 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5552ce0b-52e6-4818-aa19-ca2872fb825b-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc\" (UID: \"5552ce0b-52e6-4818-aa19-ca2872fb825b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" Sep 08 11:36:00.513184 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.513151 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5552ce0b-52e6-4818-aa19-ca2872fb825b-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc\" (UID: \"5552ce0b-52e6-4818-aa19-ca2872fb825b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" Sep 08 11:36:00.613935 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.613898 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5552ce0b-52e6-4818-aa19-ca2872fb825b-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc\" (UID: \"5552ce0b-52e6-4818-aa19-ca2872fb825b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" Sep 08 11:36:00.614113 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.613974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5552ce0b-52e6-4818-aa19-ca2872fb825b-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc\" (UID: \"5552ce0b-52e6-4818-aa19-ca2872fb825b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" Sep 08 11:36:00.614113 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.614031 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scszf\" (UniqueName: \"kubernetes.io/projected/5552ce0b-52e6-4818-aa19-ca2872fb825b-kube-api-access-scszf\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc\" (UID: \"5552ce0b-52e6-4818-aa19-ca2872fb825b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" Sep 08 11:36:00.614294 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.614273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5552ce0b-52e6-4818-aa19-ca2872fb825b-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc\" (UID: \"5552ce0b-52e6-4818-aa19-ca2872fb825b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" Sep 08 11:36:00.614361 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.614307 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5552ce0b-52e6-4818-aa19-ca2872fb825b-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc\" (UID: \"5552ce0b-52e6-4818-aa19-ca2872fb825b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" Sep 08 11:36:00.628417 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.628390 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scszf\" (UniqueName: \"kubernetes.io/projected/5552ce0b-52e6-4818-aa19-ca2872fb825b-kube-api-access-scszf\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc\" (UID: \"5552ce0b-52e6-4818-aa19-ca2872fb825b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" Sep 08 11:36:00.720184 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.720110 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" Sep 08 11:36:00.995636 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:00.995470 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc"] Sep 08 11:36:00.998288 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:36:00.998259 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5552ce0b_52e6_4818_aa19_ca2872fb825b.slice/crio-c152e3e93b1c99a15af2ed3cbe089d21f65b794d66626ddbc35de7c02ee0d51b WatchSource:0}: Error finding container c152e3e93b1c99a15af2ed3cbe089d21f65b794d66626ddbc35de7c02ee0d51b: Status 404 returned error can't find the container with id c152e3e93b1c99a15af2ed3cbe089d21f65b794d66626ddbc35de7c02ee0d51b Sep 08 11:36:01.110135 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:01.110103 2564 generic.go:356] "Generic (PLEG): container finished" podID="5552ce0b-52e6-4818-aa19-ca2872fb825b" containerID="4cecb9da92757f3d44abb9b4576f91a71917e2af1059185bbadfad0dc8897626" exitCode=0 Sep 08 11:36:01.110274 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:01.110184 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" event={"ID":"5552ce0b-52e6-4818-aa19-ca2872fb825b","Type":"ContainerDied","Data":"4cecb9da92757f3d44abb9b4576f91a71917e2af1059185bbadfad0dc8897626"} Sep 08 11:36:01.110274 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:01.110238 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" event={"ID":"5552ce0b-52e6-4818-aa19-ca2872fb825b","Type":"ContainerStarted","Data":"c152e3e93b1c99a15af2ed3cbe089d21f65b794d66626ddbc35de7c02ee0d51b"} Sep 08 11:36:02.088134 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.088096 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb"] Sep 08 11:36:02.090348 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.090332 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.097416 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.097393 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 08 11:36:02.097983 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.097966 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 08 11:36:02.099051 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.099035 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-646dc\"" Sep 08 11:36:02.099137 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.099094 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 08 11:36:02.099197 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.099177 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 08 11:36:02.117559 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.117535 2564 generic.go:356] "Generic (PLEG): container finished" podID="5552ce0b-52e6-4818-aa19-ca2872fb825b" containerID="407baa1d22149f8ecdef406db32a91a58340ad2ff9438e769f91db20a6655585" exitCode=0 Sep 08 11:36:02.117673 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.117622 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" event={"ID":"5552ce0b-52e6-4818-aa19-ca2872fb825b","Type":"ContainerDied","Data":"407baa1d22149f8ecdef406db32a91a58340ad2ff9438e769f91db20a6655585"} Sep 08 11:36:02.127991 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.127971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2033805-0e78-4a2e-afe7-d31c4fc4672d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-q7fmb\" (UID: \"c2033805-0e78-4a2e-afe7-d31c4fc4672d\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.128078 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.128000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2033805-0e78-4a2e-afe7-d31c4fc4672d-tmp\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-q7fmb\" (UID: \"c2033805-0e78-4a2e-afe7-d31c4fc4672d\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.128078 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.128023 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm4jw\" (UniqueName: \"kubernetes.io/projected/c2033805-0e78-4a2e-afe7-d31c4fc4672d-kube-api-access-fm4jw\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-q7fmb\" (UID: \"c2033805-0e78-4a2e-afe7-d31c4fc4672d\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.128158 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.128082 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2033805-0e78-4a2e-afe7-d31c4fc4672d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-q7fmb\" (UID: \"c2033805-0e78-4a2e-afe7-d31c4fc4672d\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.128158 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.128115 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2033805-0e78-4a2e-afe7-d31c4fc4672d-manifests\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-q7fmb\" (UID: \"c2033805-0e78-4a2e-afe7-d31c4fc4672d\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.174742 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.174717 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb"] Sep 08 11:36:02.229645 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.229607 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2033805-0e78-4a2e-afe7-d31c4fc4672d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-q7fmb\" (UID: \"c2033805-0e78-4a2e-afe7-d31c4fc4672d\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.229805 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.229657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2033805-0e78-4a2e-afe7-d31c4fc4672d-manifests\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-q7fmb\" (UID: \"c2033805-0e78-4a2e-afe7-d31c4fc4672d\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.229805 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.229737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2033805-0e78-4a2e-afe7-d31c4fc4672d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-q7fmb\" (UID: \"c2033805-0e78-4a2e-afe7-d31c4fc4672d\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.229805 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.229765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2033805-0e78-4a2e-afe7-d31c4fc4672d-tmp\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-q7fmb\" (UID: \"c2033805-0e78-4a2e-afe7-d31c4fc4672d\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.229805 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.229801 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm4jw\" (UniqueName: \"kubernetes.io/projected/c2033805-0e78-4a2e-afe7-d31c4fc4672d-kube-api-access-fm4jw\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-q7fmb\" (UID: \"c2033805-0e78-4a2e-afe7-d31c4fc4672d\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.230163 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.230141 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2033805-0e78-4a2e-afe7-d31c4fc4672d-manifests\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-q7fmb\" (UID: \"c2033805-0e78-4a2e-afe7-d31c4fc4672d\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.232115 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.232081 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2033805-0e78-4a2e-afe7-d31c4fc4672d-tmp\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-q7fmb\" (UID: \"c2033805-0e78-4a2e-afe7-d31c4fc4672d\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.232267 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.232248 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2033805-0e78-4a2e-afe7-d31c4fc4672d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-q7fmb\" (UID: \"c2033805-0e78-4a2e-afe7-d31c4fc4672d\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.233271 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.232569 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2033805-0e78-4a2e-afe7-d31c4fc4672d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-q7fmb\" (UID: \"c2033805-0e78-4a2e-afe7-d31c4fc4672d\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.241890 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.241867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm4jw\" (UniqueName: \"kubernetes.io/projected/c2033805-0e78-4a2e-afe7-d31c4fc4672d-kube-api-access-fm4jw\") pod \"opendatahub-operator-controller-manager-7dd45fbd48-q7fmb\" (UID: \"c2033805-0e78-4a2e-afe7-d31c4fc4672d\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.401319 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.401285 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:02.686343 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:02.686312 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb"] Sep 08 11:36:02.687547 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:36:02.687519 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2033805_0e78_4a2e_afe7_d31c4fc4672d.slice/crio-33dbaea9510d85f4a427873fb92b9b1e2a1a77e05ab14bdfccfa01e6144e6282 WatchSource:0}: Error finding container 33dbaea9510d85f4a427873fb92b9b1e2a1a77e05ab14bdfccfa01e6144e6282: Status 404 returned error can't find the container with id 33dbaea9510d85f4a427873fb92b9b1e2a1a77e05ab14bdfccfa01e6144e6282 Sep 08 11:36:03.124050 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:03.123977 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" event={"ID":"c2033805-0e78-4a2e-afe7-d31c4fc4672d","Type":"ContainerStarted","Data":"33dbaea9510d85f4a427873fb92b9b1e2a1a77e05ab14bdfccfa01e6144e6282"} Sep 08 11:36:03.126086 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:03.126065 2564 generic.go:356] "Generic (PLEG): container finished" podID="5552ce0b-52e6-4818-aa19-ca2872fb825b" containerID="f70378116da1b637e3dd2ca5666b4cac4ccc1b67a68b15d49eb6ed0882be54ab" exitCode=0 Sep 08 11:36:03.126167 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:03.126143 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" event={"ID":"5552ce0b-52e6-4818-aa19-ca2872fb825b","Type":"ContainerDied","Data":"f70378116da1b637e3dd2ca5666b4cac4ccc1b67a68b15d49eb6ed0882be54ab"} Sep 08 11:36:05.902519 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:05.902490 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" Sep 08 11:36:05.964875 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:05.964852 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scszf\" (UniqueName: \"kubernetes.io/projected/5552ce0b-52e6-4818-aa19-ca2872fb825b-kube-api-access-scszf\") pod \"5552ce0b-52e6-4818-aa19-ca2872fb825b\" (UID: \"5552ce0b-52e6-4818-aa19-ca2872fb825b\") " Sep 08 11:36:05.964973 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:05.964896 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5552ce0b-52e6-4818-aa19-ca2872fb825b-bundle\") pod \"5552ce0b-52e6-4818-aa19-ca2872fb825b\" (UID: \"5552ce0b-52e6-4818-aa19-ca2872fb825b\") " Sep 08 11:36:05.964973 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:05.964929 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5552ce0b-52e6-4818-aa19-ca2872fb825b-util\") pod \"5552ce0b-52e6-4818-aa19-ca2872fb825b\" (UID: \"5552ce0b-52e6-4818-aa19-ca2872fb825b\") " Sep 08 11:36:05.965678 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:05.965642 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5552ce0b-52e6-4818-aa19-ca2872fb825b-bundle" (OuterVolumeSpecName: "bundle") pod "5552ce0b-52e6-4818-aa19-ca2872fb825b" (UID: "5552ce0b-52e6-4818-aa19-ca2872fb825b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:36:05.966774 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:05.966750 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5552ce0b-52e6-4818-aa19-ca2872fb825b-kube-api-access-scszf" (OuterVolumeSpecName: "kube-api-access-scszf") pod "5552ce0b-52e6-4818-aa19-ca2872fb825b" (UID: "5552ce0b-52e6-4818-aa19-ca2872fb825b"). InnerVolumeSpecName "kube-api-access-scszf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:36:05.971000 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:05.970963 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5552ce0b-52e6-4818-aa19-ca2872fb825b-util" (OuterVolumeSpecName: "util") pod "5552ce0b-52e6-4818-aa19-ca2872fb825b" (UID: "5552ce0b-52e6-4818-aa19-ca2872fb825b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:36:06.066044 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:06.066016 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scszf\" (UniqueName: \"kubernetes.io/projected/5552ce0b-52e6-4818-aa19-ca2872fb825b-kube-api-access-scszf\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:36:06.066044 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:06.066040 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5552ce0b-52e6-4818-aa19-ca2872fb825b-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:36:06.066181 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:06.066051 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5552ce0b-52e6-4818-aa19-ca2872fb825b-util\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:36:06.145145 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:06.145121 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" Sep 08 11:36:06.145285 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:06.145154 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b046hcc" event={"ID":"5552ce0b-52e6-4818-aa19-ca2872fb825b","Type":"ContainerDied","Data":"c152e3e93b1c99a15af2ed3cbe089d21f65b794d66626ddbc35de7c02ee0d51b"} Sep 08 11:36:06.145285 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:06.145230 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c152e3e93b1c99a15af2ed3cbe089d21f65b794d66626ddbc35de7c02ee0d51b" Sep 08 11:36:06.147034 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:06.146976 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" event={"ID":"c2033805-0e78-4a2e-afe7-d31c4fc4672d","Type":"ContainerStarted","Data":"c5bbd93dd8b5d35ae380aaded77a9f7dce3f544dfec58327aa434025f17da87c"} Sep 08 11:36:07.153934 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:07.153902 2564 generic.go:356] "Generic (PLEG): container finished" podID="c2033805-0e78-4a2e-afe7-d31c4fc4672d" containerID="c5bbd93dd8b5d35ae380aaded77a9f7dce3f544dfec58327aa434025f17da87c" exitCode=0 Sep 08 11:36:07.154278 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:07.153971 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" event={"ID":"c2033805-0e78-4a2e-afe7-d31c4fc4672d","Type":"ContainerDied","Data":"c5bbd93dd8b5d35ae380aaded77a9f7dce3f544dfec58327aa434025f17da87c"} Sep 08 11:36:08.160260 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:08.160227 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" event={"ID":"c2033805-0e78-4a2e-afe7-d31c4fc4672d","Type":"ContainerStarted","Data":"09338d086e029ac8fb351c7f5c85a19e4bb88db40122891ae320ddb62658a077"} Sep 08 11:36:08.160685 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:08.160348 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:08.197698 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:08.197638 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" podStartSLOduration=1.427177478 podStartE2EDuration="6.197618166s" totalImagesPullingTime="3.703777237s" totalInitContainerRuntime="1.066663451s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:36:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:36:08.194086888 +0000 UTC m=+357.053274227" watchObservedRunningTime="2026-09-08 11:36:08.197618166 +0000 UTC m=+357.056805485" Sep 08 11:36:19.166776 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:19.166749 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7dd45fbd48-q7fmb" Sep 08 11:36:21.654538 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.654506 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49"] Sep 08 11:36:21.654999 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.654940 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5552ce0b-52e6-4818-aa19-ca2872fb825b" containerName="extract" Sep 08 11:36:21.654999 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.654958 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5552ce0b-52e6-4818-aa19-ca2872fb825b" containerName="extract" Sep 08 11:36:21.654999 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.654994 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5552ce0b-52e6-4818-aa19-ca2872fb825b" containerName="util" Sep 08 11:36:21.655181 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.655005 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5552ce0b-52e6-4818-aa19-ca2872fb825b" containerName="util" Sep 08 11:36:21.655181 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.655016 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5552ce0b-52e6-4818-aa19-ca2872fb825b" containerName="pull" Sep 08 11:36:21.655181 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.655024 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5552ce0b-52e6-4818-aa19-ca2872fb825b" containerName="pull" Sep 08 11:36:21.655181 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.655115 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5552ce0b-52e6-4818-aa19-ca2872fb825b" containerName="extract" Sep 08 11:36:21.672299 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.672272 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49"] Sep 08 11:36:21.673706 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.672579 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49" Sep 08 11:36:21.676593 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.676571 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 08 11:36:21.676721 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.676686 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 08 11:36:21.677749 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.677732 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 08 11:36:21.677749 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.677761 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-l2f7h\"" Sep 08 11:36:21.677960 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.677773 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 08 11:36:21.796681 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.796650 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/701c944e-55c1-4c6f-8f9e-164339efbe3d-tmp\") pod \"kube-auth-proxy-7f8d66d765-ftb49\" (UID: \"701c944e-55c1-4c6f-8f9e-164339efbe3d\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49" Sep 08 11:36:21.796842 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.796689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjkzl\" (UniqueName: \"kubernetes.io/projected/701c944e-55c1-4c6f-8f9e-164339efbe3d-kube-api-access-xjkzl\") pod \"kube-auth-proxy-7f8d66d765-ftb49\" (UID: \"701c944e-55c1-4c6f-8f9e-164339efbe3d\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49" Sep 08 11:36:21.796842 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.796714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701c944e-55c1-4c6f-8f9e-164339efbe3d-tls-certs\") pod \"kube-auth-proxy-7f8d66d765-ftb49\" (UID: \"701c944e-55c1-4c6f-8f9e-164339efbe3d\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49" Sep 08 11:36:21.897597 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.897556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701c944e-55c1-4c6f-8f9e-164339efbe3d-tls-certs\") pod \"kube-auth-proxy-7f8d66d765-ftb49\" (UID: \"701c944e-55c1-4c6f-8f9e-164339efbe3d\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49" Sep 08 11:36:21.897785 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.897669 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/701c944e-55c1-4c6f-8f9e-164339efbe3d-tmp\") pod \"kube-auth-proxy-7f8d66d765-ftb49\" (UID: \"701c944e-55c1-4c6f-8f9e-164339efbe3d\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49" Sep 08 11:36:21.897785 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:36:21.897700 2564 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 08 11:36:21.897785 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:36:21.897777 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/701c944e-55c1-4c6f-8f9e-164339efbe3d-tls-certs podName:701c944e-55c1-4c6f-8f9e-164339efbe3d nodeName:}" failed. No retries permitted until 2026-09-08 11:36:22.397760562 +0000 UTC m=+371.256947858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/701c944e-55c1-4c6f-8f9e-164339efbe3d-tls-certs") pod "kube-auth-proxy-7f8d66d765-ftb49" (UID: "701c944e-55c1-4c6f-8f9e-164339efbe3d") : secret "kube-auth-proxy-tls" not found Sep 08 11:36:21.897945 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.897706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjkzl\" (UniqueName: \"kubernetes.io/projected/701c944e-55c1-4c6f-8f9e-164339efbe3d-kube-api-access-xjkzl\") pod \"kube-auth-proxy-7f8d66d765-ftb49\" (UID: \"701c944e-55c1-4c6f-8f9e-164339efbe3d\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49" Sep 08 11:36:21.899871 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.899852 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/701c944e-55c1-4c6f-8f9e-164339efbe3d-tmp\") pod \"kube-auth-proxy-7f8d66d765-ftb49\" (UID: \"701c944e-55c1-4c6f-8f9e-164339efbe3d\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49" Sep 08 11:36:21.910862 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:21.910813 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjkzl\" (UniqueName: \"kubernetes.io/projected/701c944e-55c1-4c6f-8f9e-164339efbe3d-kube-api-access-xjkzl\") pod \"kube-auth-proxy-7f8d66d765-ftb49\" (UID: \"701c944e-55c1-4c6f-8f9e-164339efbe3d\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49" Sep 08 11:36:22.401351 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.401316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701c944e-55c1-4c6f-8f9e-164339efbe3d-tls-certs\") pod \"kube-auth-proxy-7f8d66d765-ftb49\" (UID: \"701c944e-55c1-4c6f-8f9e-164339efbe3d\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49" Sep 08 11:36:22.403769 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.403740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/701c944e-55c1-4c6f-8f9e-164339efbe3d-tls-certs\") pod \"kube-auth-proxy-7f8d66d765-ftb49\" (UID: \"701c944e-55c1-4c6f-8f9e-164339efbe3d\") " pod="openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49" Sep 08 11:36:22.584265 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.584232 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49" Sep 08 11:36:22.798231 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.798147 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft"] Sep 08 11:36:22.800852 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.800837 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:22.803640 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.803612 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 11:36:22.803754 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.803705 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 08 11:36:22.803825 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.803752 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 08 11:36:22.803897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.803879 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 08 11:36:22.804134 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.804118 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wgmlj\"" Sep 08 11:36:22.818655 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.818629 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft"] Sep 08 11:36:22.858319 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.858301 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49"] Sep 08 11:36:22.860037 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:36:22.860011 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701c944e_55c1_4c6f_8f9e_164339efbe3d.slice/crio-46e12b475ada96970ed7fbe2dce3eb09c68bfccefba33603c6b0ff8442ed9ac9 WatchSource:0}: Error finding container 46e12b475ada96970ed7fbe2dce3eb09c68bfccefba33603c6b0ff8442ed9ac9: Status 404 returned error can't find the container with id 46e12b475ada96970ed7fbe2dce3eb09c68bfccefba33603c6b0ff8442ed9ac9 Sep 08 11:36:22.905656 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.905632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:22.905782 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.905690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:22.905782 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.905725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:22.905782 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.905754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:22.905897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.905787 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:22.905897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.905859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:22.905897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:22.905892 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5cl\" (UniqueName: \"kubernetes.io/projected/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-kube-api-access-kd5cl\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.006522 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.006485 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.006663 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.006542 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.006790 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.006765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.006841 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.006812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.006935 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.006840 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.006935 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.006884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.006935 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.006902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5cl\" (UniqueName: \"kubernetes.io/projected/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-kube-api-access-kd5cl\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.007528 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.007504 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.009005 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.008977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.009117 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.009102 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.009547 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.009525 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.009623 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.009530 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.015517 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.015496 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.016008 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.015984 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5cl\" (UniqueName: \"kubernetes.io/projected/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-kube-api-access-kd5cl\") pod \"istiod-openshift-gateway-7b7995d5cd-mthft\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.110224 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.110136 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:23.234025 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.233990 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49" event={"ID":"701c944e-55c1-4c6f-8f9e-164339efbe3d","Type":"ContainerStarted","Data":"46e12b475ada96970ed7fbe2dce3eb09c68bfccefba33603c6b0ff8442ed9ac9"} Sep 08 11:36:23.385090 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:23.385066 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft"] Sep 08 11:36:23.387077 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:36:23.387049 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e59b438_1ce7_4597_aea5_cb1d1cccd5fd.slice/crio-0dbdb3592dc1812306becde7e18234791ccbb81604758b5e6c1f0ebea7b1f92f WatchSource:0}: Error finding container 0dbdb3592dc1812306becde7e18234791ccbb81604758b5e6c1f0ebea7b1f92f: Status 404 returned error can't find the container with id 0dbdb3592dc1812306becde7e18234791ccbb81604758b5e6c1f0ebea7b1f92f Sep 08 11:36:24.242695 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:24.242655 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" event={"ID":"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd","Type":"ContainerStarted","Data":"0dbdb3592dc1812306becde7e18234791ccbb81604758b5e6c1f0ebea7b1f92f"} Sep 08 11:36:26.256909 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:26.256868 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49" event={"ID":"701c944e-55c1-4c6f-8f9e-164339efbe3d","Type":"ContainerStarted","Data":"dc2f1fe9fc9eb68f044f1e6d3dcf0538172a7914a87ed93fab7f072318301723"} Sep 08 11:36:26.277249 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:26.277186 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f8d66d765-ftb49" podStartSLOduration=2.039597984 podStartE2EDuration="5.277165683s" totalImagesPullingTime="3.237567699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:36:26.275538865 +0000 UTC m=+375.134726204" watchObservedRunningTime="2026-09-08 11:36:26.277165683 +0000 UTC m=+375.136353003" Sep 08 11:36:27.026646 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:27.026600 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:36:27.026758 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:27.026691 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:36:27.263524 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:27.263485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" event={"ID":"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd","Type":"ContainerStarted","Data":"67731c0c77c68f8855ae1b446825beb341d82fdeb818e52dc0024509a1dfd4a3"} Sep 08 11:36:27.292463 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:27.292343 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" podStartSLOduration=1.655125148 podStartE2EDuration="5.292326935s" totalImagesPullingTime="3.637201787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:36:27.290358844 +0000 UTC m=+376.149546192" watchObservedRunningTime="2026-09-08 11:36:27.292326935 +0000 UTC m=+376.151514253" Sep 08 11:36:28.268833 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:28.268802 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:28.270521 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:28.270494 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-mthft container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 11:36:28.270646 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:28.270555 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" podUID="0e59b438-1ce7-4597-aea5-cb1d1cccd5fd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 11:36:29.273849 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:29.273823 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:38.880069 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:38.880031 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82"] Sep 08 11:36:38.883473 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:38.883456 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:38.898639 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:38.898615 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82"] Sep 08 11:36:38.963844 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:38.963811 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22686641-81b6-402d-b231-2e8662c60034-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:38.963999 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:38.963848 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22686641-81b6-402d-b231-2e8662c60034-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:38.963999 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:38.963872 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22686641-81b6-402d-b231-2e8662c60034-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:38.963999 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:38.963967 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22686641-81b6-402d-b231-2e8662c60034-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:38.964108 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:38.963997 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22686641-81b6-402d-b231-2e8662c60034-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:38.964108 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:38.964056 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lw7w\" (UniqueName: \"kubernetes.io/projected/22686641-81b6-402d-b231-2e8662c60034-kube-api-access-7lw7w\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:38.964169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:38.964106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22686641-81b6-402d-b231-2e8662c60034-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.064817 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.064782 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22686641-81b6-402d-b231-2e8662c60034-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.064997 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.064840 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lw7w\" (UniqueName: \"kubernetes.io/projected/22686641-81b6-402d-b231-2e8662c60034-kube-api-access-7lw7w\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.064997 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.064897 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22686641-81b6-402d-b231-2e8662c60034-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.064997 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.064950 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22686641-81b6-402d-b231-2e8662c60034-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.064997 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.064972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22686641-81b6-402d-b231-2e8662c60034-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.064997 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.064992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22686641-81b6-402d-b231-2e8662c60034-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.065265 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.065018 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22686641-81b6-402d-b231-2e8662c60034-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.065631 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.065598 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22686641-81b6-402d-b231-2e8662c60034-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.067341 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.067308 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22686641-81b6-402d-b231-2e8662c60034-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.067475 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.067356 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22686641-81b6-402d-b231-2e8662c60034-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.067475 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.067408 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22686641-81b6-402d-b231-2e8662c60034-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.067475 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.067466 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22686641-81b6-402d-b231-2e8662c60034-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.083235 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.083214 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lw7w\" (UniqueName: \"kubernetes.io/projected/22686641-81b6-402d-b231-2e8662c60034-kube-api-access-7lw7w\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.087403 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.087383 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22686641-81b6-402d-b231-2e8662c60034-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-frt82\" (UID: \"22686641-81b6-402d-b231-2e8662c60034\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.191986 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.191959 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:39.467236 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.467204 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82"] Sep 08 11:36:39.469560 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:36:39.469532 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22686641_81b6_402d_b231_2e8662c60034.slice/crio-d882c660cb7f104848a6c3131a327ed4f5a69ee3b6f792c8efc2cc9c301281fb WatchSource:0}: Error finding container d882c660cb7f104848a6c3131a327ed4f5a69ee3b6f792c8efc2cc9c301281fb: Status 404 returned error can't find the container with id d882c660cb7f104848a6c3131a327ed4f5a69ee3b6f792c8efc2cc9c301281fb Sep 08 11:36:39.472826 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.472781 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:36:39.472913 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:39.472861 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:36:40.328663 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.328628 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" event={"ID":"22686641-81b6-402d-b231-2e8662c60034","Type":"ContainerStarted","Data":"02a0c9e7270bed52c1d27c6331dca0ced40679abbcae13d5f39d34c2b4b866a9"} Sep 08 11:36:40.329094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.328677 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" event={"ID":"22686641-81b6-402d-b231-2e8662c60034","Type":"ContainerStarted","Data":"d882c660cb7f104848a6c3131a327ed4f5a69ee3b6f792c8efc2cc9c301281fb"} Sep 08 11:36:40.329094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.328785 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:40.330380 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.330348 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" Sep 08 11:36:40.362222 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.362167 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-frt82" podStartSLOduration=2.362148157 podStartE2EDuration="2.362148157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:36:40.358336175 +0000 UTC m=+389.217523521" watchObservedRunningTime="2026-09-08 11:36:40.362148157 +0000 UTC m=+389.221335477" Sep 08 11:36:40.433216 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.433176 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft"] Sep 08 11:36:40.433517 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.433486 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" podUID="0e59b438-1ce7-4597-aea5-cb1d1cccd5fd" containerName="discovery" containerID="cri-o://67731c0c77c68f8855ae1b446825beb341d82fdeb818e52dc0024509a1dfd4a3" gracePeriod=30 Sep 08 11:36:40.668870 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.668847 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:40.781553 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.781520 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-local-certs\") pod \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " Sep 08 11:36:40.781733 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.781558 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-csr-dns-cert\") pod \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " Sep 08 11:36:40.781733 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.781600 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-csr-ca-configmap\") pod \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " Sep 08 11:36:40.781733 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.781679 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-token\") pod \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " Sep 08 11:36:40.781733 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.781712 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-cacerts\") pod \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " Sep 08 11:36:40.781733 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.781732 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-kubeconfig\") pod \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " Sep 08 11:36:40.781969 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.781797 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd5cl\" (UniqueName: \"kubernetes.io/projected/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-kube-api-access-kd5cl\") pod \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\" (UID: \"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd\") " Sep 08 11:36:40.782142 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.782093 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "0e59b438-1ce7-4597-aea5-cb1d1cccd5fd" (UID: "0e59b438-1ce7-4597-aea5-cb1d1cccd5fd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:36:40.783908 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.783879 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "0e59b438-1ce7-4597-aea5-cb1d1cccd5fd" (UID: "0e59b438-1ce7-4597-aea5-cb1d1cccd5fd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:36:40.784002 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.783914 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "0e59b438-1ce7-4597-aea5-cb1d1cccd5fd" (UID: "0e59b438-1ce7-4597-aea5-cb1d1cccd5fd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:36:40.784232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.784196 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-kube-api-access-kd5cl" (OuterVolumeSpecName: "kube-api-access-kd5cl") pod "0e59b438-1ce7-4597-aea5-cb1d1cccd5fd" (UID: "0e59b438-1ce7-4597-aea5-cb1d1cccd5fd"). InnerVolumeSpecName "kube-api-access-kd5cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:36:40.784232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.784202 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-cacerts" (OuterVolumeSpecName: "cacerts") pod "0e59b438-1ce7-4597-aea5-cb1d1cccd5fd" (UID: "0e59b438-1ce7-4597-aea5-cb1d1cccd5fd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:36:40.784356 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.784272 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-local-certs" (OuterVolumeSpecName: "local-certs") pod "0e59b438-1ce7-4597-aea5-cb1d1cccd5fd" (UID: "0e59b438-1ce7-4597-aea5-cb1d1cccd5fd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:36:40.784356 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.784273 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e59b438-1ce7-4597-aea5-cb1d1cccd5fd" (UID: "0e59b438-1ce7-4597-aea5-cb1d1cccd5fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:36:40.883118 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.883060 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:36:40.883118 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.883082 2564 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-cacerts\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:36:40.883118 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.883091 2564 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-kubeconfig\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:36:40.883118 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.883099 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd5cl\" (UniqueName: \"kubernetes.io/projected/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-kube-api-access-kd5cl\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:36:40.883118 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.883109 2564 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-local-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:36:40.883118 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.883117 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-csr-dns-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:36:40.883362 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:40.883127 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd-istio-csr-ca-configmap\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:36:41.334714 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:41.334671 2564 generic.go:356] "Generic (PLEG): container finished" podID="0e59b438-1ce7-4597-aea5-cb1d1cccd5fd" containerID="67731c0c77c68f8855ae1b446825beb341d82fdeb818e52dc0024509a1dfd4a3" exitCode=0 Sep 08 11:36:41.335185 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:41.334719 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" Sep 08 11:36:41.335185 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:41.334762 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" event={"ID":"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd","Type":"ContainerDied","Data":"67731c0c77c68f8855ae1b446825beb341d82fdeb818e52dc0024509a1dfd4a3"} Sep 08 11:36:41.335185 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:41.334815 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft" event={"ID":"0e59b438-1ce7-4597-aea5-cb1d1cccd5fd","Type":"ContainerDied","Data":"0dbdb3592dc1812306becde7e18234791ccbb81604758b5e6c1f0ebea7b1f92f"} Sep 08 11:36:41.335185 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:41.334836 2564 scope.go:117] "RemoveContainer" containerID="67731c0c77c68f8855ae1b446825beb341d82fdeb818e52dc0024509a1dfd4a3" Sep 08 11:36:41.345759 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:41.345736 2564 scope.go:117] "RemoveContainer" containerID="67731c0c77c68f8855ae1b446825beb341d82fdeb818e52dc0024509a1dfd4a3" Sep 08 11:36:41.346073 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:36:41.346044 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67731c0c77c68f8855ae1b446825beb341d82fdeb818e52dc0024509a1dfd4a3\": container with ID starting with 67731c0c77c68f8855ae1b446825beb341d82fdeb818e52dc0024509a1dfd4a3 not found: ID does not exist" containerID="67731c0c77c68f8855ae1b446825beb341d82fdeb818e52dc0024509a1dfd4a3" Sep 08 11:36:41.346134 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:41.346085 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67731c0c77c68f8855ae1b446825beb341d82fdeb818e52dc0024509a1dfd4a3"} err="failed to get container status \"67731c0c77c68f8855ae1b446825beb341d82fdeb818e52dc0024509a1dfd4a3\": rpc error: code = NotFound desc = could not find container \"67731c0c77c68f8855ae1b446825beb341d82fdeb818e52dc0024509a1dfd4a3\": container with ID starting with 67731c0c77c68f8855ae1b446825beb341d82fdeb818e52dc0024509a1dfd4a3 not found: ID does not exist" Sep 08 11:36:41.364732 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:41.364706 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft"] Sep 08 11:36:41.368594 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:41.368568 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mthft"] Sep 08 11:36:41.654129 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:41.654095 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e59b438-1ce7-4597-aea5-cb1d1cccd5fd" path="/var/lib/kubelet/pods/0e59b438-1ce7-4597-aea5-cb1d1cccd5fd/volumes" Sep 08 11:36:42.013735 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.013650 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r6fln"] Sep 08 11:36:42.014052 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.014035 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e59b438-1ce7-4597-aea5-cb1d1cccd5fd" containerName="discovery" Sep 08 11:36:42.014138 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.014055 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e59b438-1ce7-4597-aea5-cb1d1cccd5fd" containerName="discovery" Sep 08 11:36:42.014194 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.014166 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e59b438-1ce7-4597-aea5-cb1d1cccd5fd" containerName="discovery" Sep 08 11:36:42.018250 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.018227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r6fln" Sep 08 11:36:42.022062 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.022041 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 08 11:36:42.022356 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.022335 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gwjns\"" Sep 08 11:36:42.037587 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.037555 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r6fln"] Sep 08 11:36:42.093239 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.093207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e91d826-a41c-4240-ab36-d8f88cf8f746-cert\") pod \"kserve-controller-manager-d6f4b65b8-r6fln\" (UID: \"6e91d826-a41c-4240-ab36-d8f88cf8f746\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r6fln" Sep 08 11:36:42.093421 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.093255 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9g57\" (UniqueName: \"kubernetes.io/projected/6e91d826-a41c-4240-ab36-d8f88cf8f746-kube-api-access-d9g57\") pod \"kserve-controller-manager-d6f4b65b8-r6fln\" (UID: \"6e91d826-a41c-4240-ab36-d8f88cf8f746\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r6fln" Sep 08 11:36:42.194636 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.194598 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e91d826-a41c-4240-ab36-d8f88cf8f746-cert\") pod \"kserve-controller-manager-d6f4b65b8-r6fln\" (UID: \"6e91d826-a41c-4240-ab36-d8f88cf8f746\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r6fln" Sep 08 11:36:42.194818 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.194666 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9g57\" (UniqueName: \"kubernetes.io/projected/6e91d826-a41c-4240-ab36-d8f88cf8f746-kube-api-access-d9g57\") pod \"kserve-controller-manager-d6f4b65b8-r6fln\" (UID: \"6e91d826-a41c-4240-ab36-d8f88cf8f746\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r6fln" Sep 08 11:36:42.197120 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.197091 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e91d826-a41c-4240-ab36-d8f88cf8f746-cert\") pod \"kserve-controller-manager-d6f4b65b8-r6fln\" (UID: \"6e91d826-a41c-4240-ab36-d8f88cf8f746\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r6fln" Sep 08 11:36:42.205954 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.205929 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9g57\" (UniqueName: \"kubernetes.io/projected/6e91d826-a41c-4240-ab36-d8f88cf8f746-kube-api-access-d9g57\") pod \"kserve-controller-manager-d6f4b65b8-r6fln\" (UID: \"6e91d826-a41c-4240-ab36-d8f88cf8f746\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r6fln" Sep 08 11:36:42.328741 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.328666 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r6fln" Sep 08 11:36:42.606458 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:42.606429 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r6fln"] Sep 08 11:36:42.608116 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:36:42.608089 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e91d826_a41c_4240_ab36_d8f88cf8f746.slice/crio-29dfd12de513bb709e104e30c41968541e22d8d053dec0b634d90863f225b097 WatchSource:0}: Error finding container 29dfd12de513bb709e104e30c41968541e22d8d053dec0b634d90863f225b097: Status 404 returned error can't find the container with id 29dfd12de513bb709e104e30c41968541e22d8d053dec0b634d90863f225b097 Sep 08 11:36:43.348508 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:43.348467 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r6fln" event={"ID":"6e91d826-a41c-4240-ab36-d8f88cf8f746","Type":"ContainerStarted","Data":"29dfd12de513bb709e104e30c41968541e22d8d053dec0b634d90863f225b097"} Sep 08 11:36:45.360805 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:45.360770 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r6fln" event={"ID":"6e91d826-a41c-4240-ab36-d8f88cf8f746","Type":"ContainerStarted","Data":"59c09333f84f2d9858e8f1d1450ff3bfd13393ff00533cab4cae00aad7bbbfca"} Sep 08 11:36:45.361215 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:45.360848 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r6fln" Sep 08 11:36:45.386974 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:36:45.386919 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r6fln" podStartSLOduration=2.156926051 podStartE2EDuration="4.386903454s" totalImagesPullingTime="2.229977403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:36:45.385709996 +0000 UTC m=+394.244897315" watchObservedRunningTime="2026-09-08 11:36:45.386903454 +0000 UTC m=+394.246090772" Sep 08 11:37:16.370215 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:16.370186 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r6fln" Sep 08 11:37:21.289339 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.289301 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f5bc5947f-l2h7n"] Sep 08 11:37:21.295660 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.295626 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.307463 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.307187 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f5bc5947f-l2h7n"] Sep 08 11:37:21.434607 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.434568 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bba36071-18ff-4c97-8b17-4706495be193-console-oauth-config\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.434768 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.434616 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bba36071-18ff-4c97-8b17-4706495be193-service-ca\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.434768 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.434719 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bba36071-18ff-4c97-8b17-4706495be193-oauth-serving-cert\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.434768 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.434752 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s9tm\" (UniqueName: \"kubernetes.io/projected/bba36071-18ff-4c97-8b17-4706495be193-kube-api-access-5s9tm\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.434985 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.434778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bba36071-18ff-4c97-8b17-4706495be193-console-serving-cert\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.434985 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.434815 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bba36071-18ff-4c97-8b17-4706495be193-console-config\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.434985 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.434926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba36071-18ff-4c97-8b17-4706495be193-trusted-ca-bundle\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.535292 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.535259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bba36071-18ff-4c97-8b17-4706495be193-oauth-serving-cert\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.535485 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.535298 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s9tm\" (UniqueName: \"kubernetes.io/projected/bba36071-18ff-4c97-8b17-4706495be193-kube-api-access-5s9tm\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.535485 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.535322 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bba36071-18ff-4c97-8b17-4706495be193-console-serving-cert\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.535485 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.535349 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bba36071-18ff-4c97-8b17-4706495be193-console-config\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.535485 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.535433 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba36071-18ff-4c97-8b17-4706495be193-trusted-ca-bundle\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.535485 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.535463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bba36071-18ff-4c97-8b17-4706495be193-console-oauth-config\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.535485 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.535485 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bba36071-18ff-4c97-8b17-4706495be193-service-ca\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.536192 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.536165 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bba36071-18ff-4c97-8b17-4706495be193-oauth-serving-cert\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.536302 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.536229 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bba36071-18ff-4c97-8b17-4706495be193-console-config\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.536302 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.536264 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bba36071-18ff-4c97-8b17-4706495be193-service-ca\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.536606 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.536408 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bba36071-18ff-4c97-8b17-4706495be193-trusted-ca-bundle\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.537848 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.537821 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bba36071-18ff-4c97-8b17-4706495be193-console-serving-cert\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.537930 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.537882 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bba36071-18ff-4c97-8b17-4706495be193-console-oauth-config\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.544719 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.544672 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s9tm\" (UniqueName: \"kubernetes.io/projected/bba36071-18ff-4c97-8b17-4706495be193-kube-api-access-5s9tm\") pod \"console-7f5bc5947f-l2h7n\" (UID: \"bba36071-18ff-4c97-8b17-4706495be193\") " pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.610345 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.610314 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:21.882782 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:21.882754 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f5bc5947f-l2h7n"] Sep 08 11:37:21.884461 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:37:21.884432 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba36071_18ff_4c97_8b17_4706495be193.slice/crio-83265e6cba54cf185da259d38d17742b2226848c42f01f690748654f69c632a5 WatchSource:0}: Error finding container 83265e6cba54cf185da259d38d17742b2226848c42f01f690748654f69c632a5: Status 404 returned error can't find the container with id 83265e6cba54cf185da259d38d17742b2226848c42f01f690748654f69c632a5 Sep 08 11:37:22.540677 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:22.540645 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5bc5947f-l2h7n" event={"ID":"bba36071-18ff-4c97-8b17-4706495be193","Type":"ContainerStarted","Data":"837e53d1da57eec541dede084209257b911d9b293297c882e4598d37702e1315"} Sep 08 11:37:22.541039 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:22.540693 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5bc5947f-l2h7n" event={"ID":"bba36071-18ff-4c97-8b17-4706495be193","Type":"ContainerStarted","Data":"83265e6cba54cf185da259d38d17742b2226848c42f01f690748654f69c632a5"} Sep 08 11:37:22.560950 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:22.560902 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f5bc5947f-l2h7n" podStartSLOduration=1.560886453 podStartE2EDuration="1.560886453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:37:22.560046747 +0000 UTC m=+431.419234076" watchObservedRunningTime="2026-09-08 11:37:22.560886453 +0000 UTC m=+431.420073774" Sep 08 11:37:31.610835 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:31.610798 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:31.611336 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:31.610882 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:31.615552 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:31.615529 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:32.592746 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:32.592716 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f5bc5947f-l2h7n" Sep 08 11:37:32.648130 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:32.648099 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-756f55cbf4-kqcs8"] Sep 08 11:37:36.810106 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.810022 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db"] Sep 08 11:37:36.813589 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.813572 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" Sep 08 11:37:36.816236 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.816215 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 11:37:36.816328 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.816254 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 11:37:36.817467 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.817449 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sjmgb\"" Sep 08 11:37:36.823501 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.823480 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db"] Sep 08 11:37:36.850963 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.850938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nms48\" (UniqueName: \"kubernetes.io/projected/caccb02b-3c02-4ff4-b0a5-d160353d42c3-kube-api-access-nms48\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db\" (UID: \"caccb02b-3c02-4ff4-b0a5-d160353d42c3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" Sep 08 11:37:36.851068 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.850991 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/caccb02b-3c02-4ff4-b0a5-d160353d42c3-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db\" (UID: \"caccb02b-3c02-4ff4-b0a5-d160353d42c3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" Sep 08 11:37:36.851068 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.851046 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/caccb02b-3c02-4ff4-b0a5-d160353d42c3-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db\" (UID: \"caccb02b-3c02-4ff4-b0a5-d160353d42c3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" Sep 08 11:37:36.907202 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.907177 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv"] Sep 08 11:37:36.910630 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.910612 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" Sep 08 11:37:36.921168 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.921144 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv"] Sep 08 11:37:36.951469 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.951443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/caccb02b-3c02-4ff4-b0a5-d160353d42c3-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db\" (UID: \"caccb02b-3c02-4ff4-b0a5-d160353d42c3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" Sep 08 11:37:36.951601 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.951487 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97f74191-9cb4-48de-9b2b-c6d1856f9380-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv\" (UID: \"97f74191-9cb4-48de-9b2b-c6d1856f9380\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" Sep 08 11:37:36.951601 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.951525 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjxcx\" (UniqueName: \"kubernetes.io/projected/97f74191-9cb4-48de-9b2b-c6d1856f9380-kube-api-access-cjxcx\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv\" (UID: \"97f74191-9cb4-48de-9b2b-c6d1856f9380\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" Sep 08 11:37:36.951601 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.951545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97f74191-9cb4-48de-9b2b-c6d1856f9380-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv\" (UID: \"97f74191-9cb4-48de-9b2b-c6d1856f9380\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" Sep 08 11:37:36.951601 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.951569 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nms48\" (UniqueName: \"kubernetes.io/projected/caccb02b-3c02-4ff4-b0a5-d160353d42c3-kube-api-access-nms48\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db\" (UID: \"caccb02b-3c02-4ff4-b0a5-d160353d42c3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" Sep 08 11:37:36.951869 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.951604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/caccb02b-3c02-4ff4-b0a5-d160353d42c3-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db\" (UID: \"caccb02b-3c02-4ff4-b0a5-d160353d42c3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" Sep 08 11:37:36.951869 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.951809 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/caccb02b-3c02-4ff4-b0a5-d160353d42c3-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db\" (UID: \"caccb02b-3c02-4ff4-b0a5-d160353d42c3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" Sep 08 11:37:36.951869 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.951851 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/caccb02b-3c02-4ff4-b0a5-d160353d42c3-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db\" (UID: \"caccb02b-3c02-4ff4-b0a5-d160353d42c3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" Sep 08 11:37:36.962830 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:36.962805 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nms48\" (UniqueName: \"kubernetes.io/projected/caccb02b-3c02-4ff4-b0a5-d160353d42c3-kube-api-access-nms48\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db\" (UID: \"caccb02b-3c02-4ff4-b0a5-d160353d42c3\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" Sep 08 11:37:37.011499 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.011468 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8"] Sep 08 11:37:37.015030 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.015012 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" Sep 08 11:37:37.024810 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.024784 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8"] Sep 08 11:37:37.052186 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.052159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjxcx\" (UniqueName: \"kubernetes.io/projected/97f74191-9cb4-48de-9b2b-c6d1856f9380-kube-api-access-cjxcx\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv\" (UID: \"97f74191-9cb4-48de-9b2b-c6d1856f9380\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" Sep 08 11:37:37.052329 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.052192 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97f74191-9cb4-48de-9b2b-c6d1856f9380-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv\" (UID: \"97f74191-9cb4-48de-9b2b-c6d1856f9380\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" Sep 08 11:37:37.052329 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.052304 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjgss\" (UniqueName: \"kubernetes.io/projected/52dba807-08c2-46c7-86e4-74df4c9fc3ec-kube-api-access-wjgss\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8\" (UID: \"52dba807-08c2-46c7-86e4-74df4c9fc3ec\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" Sep 08 11:37:37.052461 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.052403 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52dba807-08c2-46c7-86e4-74df4c9fc3ec-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8\" (UID: \"52dba807-08c2-46c7-86e4-74df4c9fc3ec\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" Sep 08 11:37:37.052526 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.052509 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97f74191-9cb4-48de-9b2b-c6d1856f9380-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv\" (UID: \"97f74191-9cb4-48de-9b2b-c6d1856f9380\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" Sep 08 11:37:37.052582 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.052541 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52dba807-08c2-46c7-86e4-74df4c9fc3ec-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8\" (UID: \"52dba807-08c2-46c7-86e4-74df4c9fc3ec\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" Sep 08 11:37:37.052582 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.052556 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97f74191-9cb4-48de-9b2b-c6d1856f9380-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv\" (UID: \"97f74191-9cb4-48de-9b2b-c6d1856f9380\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" Sep 08 11:37:37.052749 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.052732 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97f74191-9cb4-48de-9b2b-c6d1856f9380-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv\" (UID: \"97f74191-9cb4-48de-9b2b-c6d1856f9380\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" Sep 08 11:37:37.061525 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.061482 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjxcx\" (UniqueName: \"kubernetes.io/projected/97f74191-9cb4-48de-9b2b-c6d1856f9380-kube-api-access-cjxcx\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv\" (UID: \"97f74191-9cb4-48de-9b2b-c6d1856f9380\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" Sep 08 11:37:37.108154 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.108115 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf"] Sep 08 11:37:37.111726 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.111706 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" Sep 08 11:37:37.122006 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.121982 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf"] Sep 08 11:37:37.123828 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.123810 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" Sep 08 11:37:37.153249 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.153217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjgss\" (UniqueName: \"kubernetes.io/projected/52dba807-08c2-46c7-86e4-74df4c9fc3ec-kube-api-access-wjgss\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8\" (UID: \"52dba807-08c2-46c7-86e4-74df4c9fc3ec\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" Sep 08 11:37:37.153249 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.153252 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj29s\" (UniqueName: \"kubernetes.io/projected/ab697d55-0eac-47eb-857f-221e0e540f93-kube-api-access-dj29s\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf\" (UID: \"ab697d55-0eac-47eb-857f-221e0e540f93\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" Sep 08 11:37:37.153515 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.153294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52dba807-08c2-46c7-86e4-74df4c9fc3ec-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8\" (UID: \"52dba807-08c2-46c7-86e4-74df4c9fc3ec\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" Sep 08 11:37:37.153515 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.153357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab697d55-0eac-47eb-857f-221e0e540f93-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf\" (UID: \"ab697d55-0eac-47eb-857f-221e0e540f93\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" Sep 08 11:37:37.153515 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.153431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52dba807-08c2-46c7-86e4-74df4c9fc3ec-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8\" (UID: \"52dba807-08c2-46c7-86e4-74df4c9fc3ec\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" Sep 08 11:37:37.153515 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.153462 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab697d55-0eac-47eb-857f-221e0e540f93-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf\" (UID: \"ab697d55-0eac-47eb-857f-221e0e540f93\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" Sep 08 11:37:37.153723 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.153656 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52dba807-08c2-46c7-86e4-74df4c9fc3ec-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8\" (UID: \"52dba807-08c2-46c7-86e4-74df4c9fc3ec\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" Sep 08 11:37:37.153777 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.153754 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52dba807-08c2-46c7-86e4-74df4c9fc3ec-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8\" (UID: \"52dba807-08c2-46c7-86e4-74df4c9fc3ec\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" Sep 08 11:37:37.163293 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.163262 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjgss\" (UniqueName: \"kubernetes.io/projected/52dba807-08c2-46c7-86e4-74df4c9fc3ec-kube-api-access-wjgss\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8\" (UID: \"52dba807-08c2-46c7-86e4-74df4c9fc3ec\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" Sep 08 11:37:37.219829 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.219798 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" Sep 08 11:37:37.254202 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.254168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab697d55-0eac-47eb-857f-221e0e540f93-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf\" (UID: \"ab697d55-0eac-47eb-857f-221e0e540f93\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" Sep 08 11:37:37.254345 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.254231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj29s\" (UniqueName: \"kubernetes.io/projected/ab697d55-0eac-47eb-857f-221e0e540f93-kube-api-access-dj29s\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf\" (UID: \"ab697d55-0eac-47eb-857f-221e0e540f93\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" Sep 08 11:37:37.254345 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.254278 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab697d55-0eac-47eb-857f-221e0e540f93-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf\" (UID: \"ab697d55-0eac-47eb-857f-221e0e540f93\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" Sep 08 11:37:37.254635 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.254613 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab697d55-0eac-47eb-857f-221e0e540f93-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf\" (UID: \"ab697d55-0eac-47eb-857f-221e0e540f93\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" Sep 08 11:37:37.254677 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.254635 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab697d55-0eac-47eb-857f-221e0e540f93-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf\" (UID: \"ab697d55-0eac-47eb-857f-221e0e540f93\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" Sep 08 11:37:37.264504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.264474 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj29s\" (UniqueName: \"kubernetes.io/projected/ab697d55-0eac-47eb-857f-221e0e540f93-kube-api-access-dj29s\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf\" (UID: \"ab697d55-0eac-47eb-857f-221e0e540f93\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" Sep 08 11:37:37.324858 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.324780 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" Sep 08 11:37:37.403974 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.403940 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db"] Sep 08 11:37:37.404983 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:37:37.404952 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaccb02b_3c02_4ff4_b0a5_d160353d42c3.slice/crio-843387b5e7ded0aebea30baa1f4f21ea5d76f164df696e372cbaad6527477471 WatchSource:0}: Error finding container 843387b5e7ded0aebea30baa1f4f21ea5d76f164df696e372cbaad6527477471: Status 404 returned error can't find the container with id 843387b5e7ded0aebea30baa1f4f21ea5d76f164df696e372cbaad6527477471 Sep 08 11:37:37.425576 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.421849 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" Sep 08 11:37:37.494537 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.494479 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv"] Sep 08 11:37:37.498951 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:37:37.498917 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f74191_9cb4_48de_9b2b_c6d1856f9380.slice/crio-e0d68bc7df5c3ed9ed609b5c5bb10d04278f8730cf7cb84eb1d733295ba9c031 WatchSource:0}: Error finding container e0d68bc7df5c3ed9ed609b5c5bb10d04278f8730cf7cb84eb1d733295ba9c031: Status 404 returned error can't find the container with id e0d68bc7df5c3ed9ed609b5c5bb10d04278f8730cf7cb84eb1d733295ba9c031 Sep 08 11:37:37.603475 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.603411 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8"] Sep 08 11:37:37.605338 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:37:37.605313 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52dba807_08c2_46c7_86e4_74df4c9fc3ec.slice/crio-1b8d48201f2a83907233bf1457bdf3efdf76f5b149f4d070c09bf3242ab7e6e6 WatchSource:0}: Error finding container 1b8d48201f2a83907233bf1457bdf3efdf76f5b149f4d070c09bf3242ab7e6e6: Status 404 returned error can't find the container with id 1b8d48201f2a83907233bf1457bdf3efdf76f5b149f4d070c09bf3242ab7e6e6 Sep 08 11:37:37.616677 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.616648 2564 generic.go:356] "Generic (PLEG): container finished" podID="caccb02b-3c02-4ff4-b0a5-d160353d42c3" containerID="b8776c89007f8f895924fea257f035ce6b5698dd47132d83744826ba4e219ee1" exitCode=0 Sep 08 11:37:37.616773 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.616722 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" event={"ID":"caccb02b-3c02-4ff4-b0a5-d160353d42c3","Type":"ContainerDied","Data":"b8776c89007f8f895924fea257f035ce6b5698dd47132d83744826ba4e219ee1"} Sep 08 11:37:37.616773 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.616769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" event={"ID":"caccb02b-3c02-4ff4-b0a5-d160353d42c3","Type":"ContainerStarted","Data":"843387b5e7ded0aebea30baa1f4f21ea5d76f164df696e372cbaad6527477471"} Sep 08 11:37:37.618446 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.618419 2564 generic.go:356] "Generic (PLEG): container finished" podID="97f74191-9cb4-48de-9b2b-c6d1856f9380" containerID="99362de4fc0c77fb9e27bc7ac5c7d78b15a6fe0a7fa5ca9793457cc039f3c6f8" exitCode=0 Sep 08 11:37:37.618720 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.618541 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" event={"ID":"97f74191-9cb4-48de-9b2b-c6d1856f9380","Type":"ContainerDied","Data":"99362de4fc0c77fb9e27bc7ac5c7d78b15a6fe0a7fa5ca9793457cc039f3c6f8"} Sep 08 11:37:37.618720 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.618593 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" event={"ID":"97f74191-9cb4-48de-9b2b-c6d1856f9380","Type":"ContainerStarted","Data":"e0d68bc7df5c3ed9ed609b5c5bb10d04278f8730cf7cb84eb1d733295ba9c031"} Sep 08 11:37:37.620097 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.620070 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" event={"ID":"52dba807-08c2-46c7-86e4-74df4c9fc3ec","Type":"ContainerStarted","Data":"1b8d48201f2a83907233bf1457bdf3efdf76f5b149f4d070c09bf3242ab7e6e6"} Sep 08 11:37:37.707314 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:37.707287 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf"] Sep 08 11:37:37.709070 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:37:37.709048 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab697d55_0eac_47eb_857f_221e0e540f93.slice/crio-ca76cd76f99e41c500dd54c6b6301c4422bc16a72799edc0926ed7e2e4804a4d WatchSource:0}: Error finding container ca76cd76f99e41c500dd54c6b6301c4422bc16a72799edc0926ed7e2e4804a4d: Status 404 returned error can't find the container with id ca76cd76f99e41c500dd54c6b6301c4422bc16a72799edc0926ed7e2e4804a4d Sep 08 11:37:38.626221 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:38.626194 2564 generic.go:356] "Generic (PLEG): container finished" podID="52dba807-08c2-46c7-86e4-74df4c9fc3ec" containerID="7c2cc0d59483a3dd08945008311572bf88afab942e64c4b03e63b82940918c08" exitCode=0 Sep 08 11:37:38.626587 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:38.626262 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" event={"ID":"52dba807-08c2-46c7-86e4-74df4c9fc3ec","Type":"ContainerDied","Data":"7c2cc0d59483a3dd08945008311572bf88afab942e64c4b03e63b82940918c08"} Sep 08 11:37:38.628285 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:38.628259 2564 generic.go:356] "Generic (PLEG): container finished" podID="ab697d55-0eac-47eb-857f-221e0e540f93" containerID="b20b1273c38f5c02919b06f098f4e48b9c2b1edbc02814d17c7c2ea1aa7b3f9f" exitCode=0 Sep 08 11:37:38.628392 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:38.628338 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" event={"ID":"ab697d55-0eac-47eb-857f-221e0e540f93","Type":"ContainerDied","Data":"b20b1273c38f5c02919b06f098f4e48b9c2b1edbc02814d17c7c2ea1aa7b3f9f"} Sep 08 11:37:38.628443 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:38.628393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" event={"ID":"ab697d55-0eac-47eb-857f-221e0e540f93","Type":"ContainerStarted","Data":"ca76cd76f99e41c500dd54c6b6301c4422bc16a72799edc0926ed7e2e4804a4d"} Sep 08 11:37:39.634740 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:39.634712 2564 generic.go:356] "Generic (PLEG): container finished" podID="97f74191-9cb4-48de-9b2b-c6d1856f9380" containerID="4d069d85e325643d1970b0b9ea437672bb8a586fda29cf17769c4be378b72ec5" exitCode=0 Sep 08 11:37:39.635062 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:39.634795 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" event={"ID":"97f74191-9cb4-48de-9b2b-c6d1856f9380","Type":"ContainerDied","Data":"4d069d85e325643d1970b0b9ea437672bb8a586fda29cf17769c4be378b72ec5"} Sep 08 11:37:39.636815 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:39.636795 2564 generic.go:356] "Generic (PLEG): container finished" podID="caccb02b-3c02-4ff4-b0a5-d160353d42c3" containerID="e0d19525d638070edb77d85bd96e450b5deb1192aa1dd3eba5cf90989e9a4566" exitCode=0 Sep 08 11:37:39.636917 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:39.636875 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" event={"ID":"caccb02b-3c02-4ff4-b0a5-d160353d42c3","Type":"ContainerDied","Data":"e0d19525d638070edb77d85bd96e450b5deb1192aa1dd3eba5cf90989e9a4566"} Sep 08 11:37:40.643978 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:40.643946 2564 generic.go:356] "Generic (PLEG): container finished" podID="97f74191-9cb4-48de-9b2b-c6d1856f9380" containerID="3d0490fadf8022770a19bce4c1bbe0781da72b7d3e667ef9a0c51cd979e7622e" exitCode=0 Sep 08 11:37:40.644402 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:40.644036 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" event={"ID":"97f74191-9cb4-48de-9b2b-c6d1856f9380","Type":"ContainerDied","Data":"3d0490fadf8022770a19bce4c1bbe0781da72b7d3e667ef9a0c51cd979e7622e"} Sep 08 11:37:40.645924 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:40.645903 2564 generic.go:356] "Generic (PLEG): container finished" podID="52dba807-08c2-46c7-86e4-74df4c9fc3ec" containerID="408cfcb94c91a85abc6af23a3ceddc196373d57cd5ea42b75aceb2e9594422de" exitCode=0 Sep 08 11:37:40.646023 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:40.645969 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" event={"ID":"52dba807-08c2-46c7-86e4-74df4c9fc3ec","Type":"ContainerDied","Data":"408cfcb94c91a85abc6af23a3ceddc196373d57cd5ea42b75aceb2e9594422de"} Sep 08 11:37:40.648297 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:40.648275 2564 generic.go:356] "Generic (PLEG): container finished" podID="caccb02b-3c02-4ff4-b0a5-d160353d42c3" containerID="bf10a0fd4ca75fe155797086190a772797c7ce82a50da85afe1f36a63d0bf40a" exitCode=0 Sep 08 11:37:40.648402 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:40.648350 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" event={"ID":"caccb02b-3c02-4ff4-b0a5-d160353d42c3","Type":"ContainerDied","Data":"bf10a0fd4ca75fe155797086190a772797c7ce82a50da85afe1f36a63d0bf40a"} Sep 08 11:37:40.650049 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:40.650031 2564 generic.go:356] "Generic (PLEG): container finished" podID="ab697d55-0eac-47eb-857f-221e0e540f93" containerID="5efb31ab64a919b728e53513dfe86ddb39604c1a7292b6f16cde73aa47b72e32" exitCode=0 Sep 08 11:37:40.650141 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:40.650078 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" event={"ID":"ab697d55-0eac-47eb-857f-221e0e540f93","Type":"ContainerDied","Data":"5efb31ab64a919b728e53513dfe86ddb39604c1a7292b6f16cde73aa47b72e32"} Sep 08 11:37:41.662609 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.662579 2564 generic.go:356] "Generic (PLEG): container finished" podID="52dba807-08c2-46c7-86e4-74df4c9fc3ec" containerID="6c83334612a6c33002a64fdfe20dc4f93ee324f3452f5a6754dc2227d366d081" exitCode=0 Sep 08 11:37:41.663090 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.662660 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" event={"ID":"52dba807-08c2-46c7-86e4-74df4c9fc3ec","Type":"ContainerDied","Data":"6c83334612a6c33002a64fdfe20dc4f93ee324f3452f5a6754dc2227d366d081"} Sep 08 11:37:41.664723 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.664698 2564 generic.go:356] "Generic (PLEG): container finished" podID="ab697d55-0eac-47eb-857f-221e0e540f93" containerID="3afe08774ffbe1e5d529aaa918b6838c6867b87e9f729b2cae8470e8808a6dbb" exitCode=0 Sep 08 11:37:41.664839 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.664765 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" event={"ID":"ab697d55-0eac-47eb-857f-221e0e540f93","Type":"ContainerDied","Data":"3afe08774ffbe1e5d529aaa918b6838c6867b87e9f729b2cae8470e8808a6dbb"} Sep 08 11:37:41.796138 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.796115 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" Sep 08 11:37:41.833197 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.833170 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" Sep 08 11:37:41.890255 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.890223 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97f74191-9cb4-48de-9b2b-c6d1856f9380-bundle\") pod \"97f74191-9cb4-48de-9b2b-c6d1856f9380\" (UID: \"97f74191-9cb4-48de-9b2b-c6d1856f9380\") " Sep 08 11:37:41.890415 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.890269 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/caccb02b-3c02-4ff4-b0a5-d160353d42c3-util\") pod \"caccb02b-3c02-4ff4-b0a5-d160353d42c3\" (UID: \"caccb02b-3c02-4ff4-b0a5-d160353d42c3\") " Sep 08 11:37:41.890415 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.890293 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nms48\" (UniqueName: \"kubernetes.io/projected/caccb02b-3c02-4ff4-b0a5-d160353d42c3-kube-api-access-nms48\") pod \"caccb02b-3c02-4ff4-b0a5-d160353d42c3\" (UID: \"caccb02b-3c02-4ff4-b0a5-d160353d42c3\") " Sep 08 11:37:41.890415 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.890311 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjxcx\" (UniqueName: \"kubernetes.io/projected/97f74191-9cb4-48de-9b2b-c6d1856f9380-kube-api-access-cjxcx\") pod \"97f74191-9cb4-48de-9b2b-c6d1856f9380\" (UID: \"97f74191-9cb4-48de-9b2b-c6d1856f9380\") " Sep 08 11:37:41.890588 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.890363 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/caccb02b-3c02-4ff4-b0a5-d160353d42c3-bundle\") pod \"caccb02b-3c02-4ff4-b0a5-d160353d42c3\" (UID: \"caccb02b-3c02-4ff4-b0a5-d160353d42c3\") " Sep 08 11:37:41.890588 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.890448 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97f74191-9cb4-48de-9b2b-c6d1856f9380-util\") pod \"97f74191-9cb4-48de-9b2b-c6d1856f9380\" (UID: \"97f74191-9cb4-48de-9b2b-c6d1856f9380\") " Sep 08 11:37:41.890891 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.890858 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f74191-9cb4-48de-9b2b-c6d1856f9380-bundle" (OuterVolumeSpecName: "bundle") pod "97f74191-9cb4-48de-9b2b-c6d1856f9380" (UID: "97f74191-9cb4-48de-9b2b-c6d1856f9380"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:37:41.891031 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.890987 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caccb02b-3c02-4ff4-b0a5-d160353d42c3-bundle" (OuterVolumeSpecName: "bundle") pod "caccb02b-3c02-4ff4-b0a5-d160353d42c3" (UID: "caccb02b-3c02-4ff4-b0a5-d160353d42c3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:37:41.893026 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.892997 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caccb02b-3c02-4ff4-b0a5-d160353d42c3-kube-api-access-nms48" (OuterVolumeSpecName: "kube-api-access-nms48") pod "caccb02b-3c02-4ff4-b0a5-d160353d42c3" (UID: "caccb02b-3c02-4ff4-b0a5-d160353d42c3"). InnerVolumeSpecName "kube-api-access-nms48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:37:41.893126 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.893005 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f74191-9cb4-48de-9b2b-c6d1856f9380-kube-api-access-cjxcx" (OuterVolumeSpecName: "kube-api-access-cjxcx") pod "97f74191-9cb4-48de-9b2b-c6d1856f9380" (UID: "97f74191-9cb4-48de-9b2b-c6d1856f9380"). InnerVolumeSpecName "kube-api-access-cjxcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:37:41.895987 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.895964 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f74191-9cb4-48de-9b2b-c6d1856f9380-util" (OuterVolumeSpecName: "util") pod "97f74191-9cb4-48de-9b2b-c6d1856f9380" (UID: "97f74191-9cb4-48de-9b2b-c6d1856f9380"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:37:41.896210 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.896191 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caccb02b-3c02-4ff4-b0a5-d160353d42c3-util" (OuterVolumeSpecName: "util") pod "caccb02b-3c02-4ff4-b0a5-d160353d42c3" (UID: "caccb02b-3c02-4ff4-b0a5-d160353d42c3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:37:41.991749 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.991690 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/caccb02b-3c02-4ff4-b0a5-d160353d42c3-util\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:41.991749 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.991730 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nms48\" (UniqueName: \"kubernetes.io/projected/caccb02b-3c02-4ff4-b0a5-d160353d42c3-kube-api-access-nms48\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:41.991749 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.991741 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjxcx\" (UniqueName: \"kubernetes.io/projected/97f74191-9cb4-48de-9b2b-c6d1856f9380-kube-api-access-cjxcx\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:41.991749 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.991751 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/caccb02b-3c02-4ff4-b0a5-d160353d42c3-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:41.991948 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.991759 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97f74191-9cb4-48de-9b2b-c6d1856f9380-util\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:41.991948 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:41.991766 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97f74191-9cb4-48de-9b2b-c6d1856f9380-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:42.671122 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.671095 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" Sep 08 11:37:42.671122 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.671114 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0ctkgv" event={"ID":"97f74191-9cb4-48de-9b2b-c6d1856f9380","Type":"ContainerDied","Data":"e0d68bc7df5c3ed9ed609b5c5bb10d04278f8730cf7cb84eb1d733295ba9c031"} Sep 08 11:37:42.671584 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.671151 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0d68bc7df5c3ed9ed609b5c5bb10d04278f8730cf7cb84eb1d733295ba9c031" Sep 08 11:37:42.673213 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.673193 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" Sep 08 11:37:42.673354 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.673223 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c5c4db" event={"ID":"caccb02b-3c02-4ff4-b0a5-d160353d42c3","Type":"ContainerDied","Data":"843387b5e7ded0aebea30baa1f4f21ea5d76f164df696e372cbaad6527477471"} Sep 08 11:37:42.673354 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.673258 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="843387b5e7ded0aebea30baa1f4f21ea5d76f164df696e372cbaad6527477471" Sep 08 11:37:42.842020 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.841995 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" Sep 08 11:37:42.844137 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.844118 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" Sep 08 11:37:42.897782 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.897756 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab697d55-0eac-47eb-857f-221e0e540f93-util\") pod \"ab697d55-0eac-47eb-857f-221e0e540f93\" (UID: \"ab697d55-0eac-47eb-857f-221e0e540f93\") " Sep 08 11:37:42.897938 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.897804 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjgss\" (UniqueName: \"kubernetes.io/projected/52dba807-08c2-46c7-86e4-74df4c9fc3ec-kube-api-access-wjgss\") pod \"52dba807-08c2-46c7-86e4-74df4c9fc3ec\" (UID: \"52dba807-08c2-46c7-86e4-74df4c9fc3ec\") " Sep 08 11:37:42.897938 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.897820 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52dba807-08c2-46c7-86e4-74df4c9fc3ec-util\") pod \"52dba807-08c2-46c7-86e4-74df4c9fc3ec\" (UID: \"52dba807-08c2-46c7-86e4-74df4c9fc3ec\") " Sep 08 11:37:42.897938 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.897860 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj29s\" (UniqueName: \"kubernetes.io/projected/ab697d55-0eac-47eb-857f-221e0e540f93-kube-api-access-dj29s\") pod \"ab697d55-0eac-47eb-857f-221e0e540f93\" (UID: \"ab697d55-0eac-47eb-857f-221e0e540f93\") " Sep 08 11:37:42.897938 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.897883 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52dba807-08c2-46c7-86e4-74df4c9fc3ec-bundle\") pod \"52dba807-08c2-46c7-86e4-74df4c9fc3ec\" (UID: \"52dba807-08c2-46c7-86e4-74df4c9fc3ec\") " Sep 08 11:37:42.897938 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.897914 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab697d55-0eac-47eb-857f-221e0e540f93-bundle\") pod \"ab697d55-0eac-47eb-857f-221e0e540f93\" (UID: \"ab697d55-0eac-47eb-857f-221e0e540f93\") " Sep 08 11:37:42.898534 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.898505 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52dba807-08c2-46c7-86e4-74df4c9fc3ec-bundle" (OuterVolumeSpecName: "bundle") pod "52dba807-08c2-46c7-86e4-74df4c9fc3ec" (UID: "52dba807-08c2-46c7-86e4-74df4c9fc3ec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:37:42.898879 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.898858 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab697d55-0eac-47eb-857f-221e0e540f93-bundle" (OuterVolumeSpecName: "bundle") pod "ab697d55-0eac-47eb-857f-221e0e540f93" (UID: "ab697d55-0eac-47eb-857f-221e0e540f93"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:37:42.900620 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.900592 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52dba807-08c2-46c7-86e4-74df4c9fc3ec-kube-api-access-wjgss" (OuterVolumeSpecName: "kube-api-access-wjgss") pod "52dba807-08c2-46c7-86e4-74df4c9fc3ec" (UID: "52dba807-08c2-46c7-86e4-74df4c9fc3ec"). InnerVolumeSpecName "kube-api-access-wjgss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:37:42.900717 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.900695 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab697d55-0eac-47eb-857f-221e0e540f93-kube-api-access-dj29s" (OuterVolumeSpecName: "kube-api-access-dj29s") pod "ab697d55-0eac-47eb-857f-221e0e540f93" (UID: "ab697d55-0eac-47eb-857f-221e0e540f93"). InnerVolumeSpecName "kube-api-access-dj29s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:37:42.904115 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.904091 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52dba807-08c2-46c7-86e4-74df4c9fc3ec-util" (OuterVolumeSpecName: "util") pod "52dba807-08c2-46c7-86e4-74df4c9fc3ec" (UID: "52dba807-08c2-46c7-86e4-74df4c9fc3ec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:37:42.905060 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.905037 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab697d55-0eac-47eb-857f-221e0e540f93-util" (OuterVolumeSpecName: "util") pod "ab697d55-0eac-47eb-857f-221e0e540f93" (UID: "ab697d55-0eac-47eb-857f-221e0e540f93"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:37:42.999504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.999428 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab697d55-0eac-47eb-857f-221e0e540f93-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:42.999504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.999452 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab697d55-0eac-47eb-857f-221e0e540f93-util\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:42.999504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.999465 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjgss\" (UniqueName: \"kubernetes.io/projected/52dba807-08c2-46c7-86e4-74df4c9fc3ec-kube-api-access-wjgss\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:42.999504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.999479 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52dba807-08c2-46c7-86e4-74df4c9fc3ec-util\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:42.999504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.999492 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj29s\" (UniqueName: \"kubernetes.io/projected/ab697d55-0eac-47eb-857f-221e0e540f93-kube-api-access-dj29s\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:42.999504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:42.999505 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52dba807-08c2-46c7-86e4-74df4c9fc3ec-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:43.679763 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:43.679742 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" Sep 08 11:37:43.680094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:43.679774 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrc8q8" event={"ID":"52dba807-08c2-46c7-86e4-74df4c9fc3ec","Type":"ContainerDied","Data":"1b8d48201f2a83907233bf1457bdf3efdf76f5b149f4d070c09bf3242ab7e6e6"} Sep 08 11:37:43.680094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:43.679816 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b8d48201f2a83907233bf1457bdf3efdf76f5b149f4d070c09bf3242ab7e6e6" Sep 08 11:37:43.682022 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:43.682000 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" event={"ID":"ab697d55-0eac-47eb-857f-221e0e540f93","Type":"ContainerDied","Data":"ca76cd76f99e41c500dd54c6b6301c4422bc16a72799edc0926ed7e2e4804a4d"} Sep 08 11:37:43.682140 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:43.682032 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca76cd76f99e41c500dd54c6b6301c4422bc16a72799edc0926ed7e2e4804a4d" Sep 08 11:37:43.682226 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:43.682210 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534z5nf" Sep 08 11:37:50.728713 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.728680 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v"] Sep 08 11:37:50.729074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.728990 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caccb02b-3c02-4ff4-b0a5-d160353d42c3" containerName="pull" Sep 08 11:37:50.729074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729001 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="caccb02b-3c02-4ff4-b0a5-d160353d42c3" containerName="pull" Sep 08 11:37:50.729074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729009 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97f74191-9cb4-48de-9b2b-c6d1856f9380" containerName="extract" Sep 08 11:37:50.729074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729015 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97f74191-9cb4-48de-9b2b-c6d1856f9380" containerName="extract" Sep 08 11:37:50.729074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729021 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caccb02b-3c02-4ff4-b0a5-d160353d42c3" containerName="extract" Sep 08 11:37:50.729074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729027 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="caccb02b-3c02-4ff4-b0a5-d160353d42c3" containerName="extract" Sep 08 11:37:50.729074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729034 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97f74191-9cb4-48de-9b2b-c6d1856f9380" containerName="pull" Sep 08 11:37:50.729074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729039 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97f74191-9cb4-48de-9b2b-c6d1856f9380" containerName="pull" Sep 08 11:37:50.729074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729046 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab697d55-0eac-47eb-857f-221e0e540f93" containerName="pull" Sep 08 11:37:50.729074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729051 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab697d55-0eac-47eb-857f-221e0e540f93" containerName="pull" Sep 08 11:37:50.729074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729059 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97f74191-9cb4-48de-9b2b-c6d1856f9380" containerName="util" Sep 08 11:37:50.729074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729063 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97f74191-9cb4-48de-9b2b-c6d1856f9380" containerName="util" Sep 08 11:37:50.729074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729068 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab697d55-0eac-47eb-857f-221e0e540f93" containerName="util" Sep 08 11:37:50.729074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729074 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab697d55-0eac-47eb-857f-221e0e540f93" containerName="util" Sep 08 11:37:50.729074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729079 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52dba807-08c2-46c7-86e4-74df4c9fc3ec" containerName="pull" Sep 08 11:37:50.729536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729084 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="52dba807-08c2-46c7-86e4-74df4c9fc3ec" containerName="pull" Sep 08 11:37:50.729536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729093 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52dba807-08c2-46c7-86e4-74df4c9fc3ec" containerName="extract" Sep 08 11:37:50.729536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729099 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="52dba807-08c2-46c7-86e4-74df4c9fc3ec" containerName="extract" Sep 08 11:37:50.729536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729104 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caccb02b-3c02-4ff4-b0a5-d160353d42c3" containerName="util" Sep 08 11:37:50.729536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729109 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="caccb02b-3c02-4ff4-b0a5-d160353d42c3" containerName="util" Sep 08 11:37:50.729536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729116 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52dba807-08c2-46c7-86e4-74df4c9fc3ec" containerName="util" Sep 08 11:37:50.729536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729121 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="52dba807-08c2-46c7-86e4-74df4c9fc3ec" containerName="util" Sep 08 11:37:50.729536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729126 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab697d55-0eac-47eb-857f-221e0e540f93" containerName="extract" Sep 08 11:37:50.729536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729131 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab697d55-0eac-47eb-857f-221e0e540f93" containerName="extract" Sep 08 11:37:50.729536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729185 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="52dba807-08c2-46c7-86e4-74df4c9fc3ec" containerName="extract" Sep 08 11:37:50.729536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729192 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="caccb02b-3c02-4ff4-b0a5-d160353d42c3" containerName="extract" Sep 08 11:37:50.729536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729198 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97f74191-9cb4-48de-9b2b-c6d1856f9380" containerName="extract" Sep 08 11:37:50.729536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.729204 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab697d55-0eac-47eb-857f-221e0e540f93" containerName="extract" Sep 08 11:37:50.733246 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.733230 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" Sep 08 11:37:50.736362 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.736335 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-d9c72\"" Sep 08 11:37:50.736984 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.736964 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 08 11:37:50.737091 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.736992 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 08 11:37:50.743945 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.743922 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v"] Sep 08 11:37:50.765052 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.765025 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aad12b53-e462-450b-a782-e2a047bae7a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-tjw8v\" (UID: \"aad12b53-e462-450b-a782-e2a047bae7a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" Sep 08 11:37:50.765139 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.765061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfwdc\" (UniqueName: \"kubernetes.io/projected/aad12b53-e462-450b-a782-e2a047bae7a6-kube-api-access-xfwdc\") pod \"kuadrant-operator-controller-manager-58fc56757c-tjw8v\" (UID: \"aad12b53-e462-450b-a782-e2a047bae7a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" Sep 08 11:37:50.866280 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.866245 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aad12b53-e462-450b-a782-e2a047bae7a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-tjw8v\" (UID: \"aad12b53-e462-450b-a782-e2a047bae7a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" Sep 08 11:37:50.866280 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.866289 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfwdc\" (UniqueName: \"kubernetes.io/projected/aad12b53-e462-450b-a782-e2a047bae7a6-kube-api-access-xfwdc\") pod \"kuadrant-operator-controller-manager-58fc56757c-tjw8v\" (UID: \"aad12b53-e462-450b-a782-e2a047bae7a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" Sep 08 11:37:50.866638 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.866619 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aad12b53-e462-450b-a782-e2a047bae7a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-tjw8v\" (UID: \"aad12b53-e462-450b-a782-e2a047bae7a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" Sep 08 11:37:50.881843 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:50.881817 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfwdc\" (UniqueName: \"kubernetes.io/projected/aad12b53-e462-450b-a782-e2a047bae7a6-kube-api-access-xfwdc\") pod \"kuadrant-operator-controller-manager-58fc56757c-tjw8v\" (UID: \"aad12b53-e462-450b-a782-e2a047bae7a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" Sep 08 11:37:51.045298 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:51.045189 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" Sep 08 11:37:51.332697 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:51.332668 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v"] Sep 08 11:37:51.334164 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:37:51.334141 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad12b53_e462_450b_a782_e2a047bae7a6.slice/crio-c107aedf27c01abc2af97f54728342206671d55bc945111f319144825347b2ff WatchSource:0}: Error finding container c107aedf27c01abc2af97f54728342206671d55bc945111f319144825347b2ff: Status 404 returned error can't find the container with id c107aedf27c01abc2af97f54728342206671d55bc945111f319144825347b2ff Sep 08 11:37:51.722498 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:51.722460 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" event={"ID":"aad12b53-e462-450b-a782-e2a047bae7a6","Type":"ContainerStarted","Data":"c107aedf27c01abc2af97f54728342206671d55bc945111f319144825347b2ff"} Sep 08 11:37:54.964337 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:54.964301 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-hrnxj"] Sep 08 11:37:54.968193 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:54.968173 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hrnxj" Sep 08 11:37:54.970843 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:54.970819 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 08 11:37:54.970935 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:54.970852 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fm8nk\"" Sep 08 11:37:54.979143 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:54.979121 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-hrnxj"] Sep 08 11:37:55.003605 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:55.003573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2rkw\" (UniqueName: \"kubernetes.io/projected/00cbf4b6-5a38-46f2-9b62-7a5d4088bc9e-kube-api-access-k2rkw\") pod \"dns-operator-controller-manager-5985564776-hrnxj\" (UID: \"00cbf4b6-5a38-46f2-9b62-7a5d4088bc9e\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-hrnxj" Sep 08 11:37:55.104797 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:55.104764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2rkw\" (UniqueName: \"kubernetes.io/projected/00cbf4b6-5a38-46f2-9b62-7a5d4088bc9e-kube-api-access-k2rkw\") pod \"dns-operator-controller-manager-5985564776-hrnxj\" (UID: \"00cbf4b6-5a38-46f2-9b62-7a5d4088bc9e\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-hrnxj" Sep 08 11:37:55.116538 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:55.116515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2rkw\" (UniqueName: \"kubernetes.io/projected/00cbf4b6-5a38-46f2-9b62-7a5d4088bc9e-kube-api-access-k2rkw\") pod \"dns-operator-controller-manager-5985564776-hrnxj\" (UID: \"00cbf4b6-5a38-46f2-9b62-7a5d4088bc9e\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-hrnxj" Sep 08 11:37:55.279797 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:55.279713 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hrnxj" Sep 08 11:37:55.555867 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:55.555843 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-hrnxj"] Sep 08 11:37:55.557875 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:37:55.557847 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00cbf4b6_5a38_46f2_9b62_7a5d4088bc9e.slice/crio-57e47f7dab3074e890498b450752fc69e386a56ba9485ffd9a0bd13712e98844 WatchSource:0}: Error finding container 57e47f7dab3074e890498b450752fc69e386a56ba9485ffd9a0bd13712e98844: Status 404 returned error can't find the container with id 57e47f7dab3074e890498b450752fc69e386a56ba9485ffd9a0bd13712e98844 Sep 08 11:37:55.746823 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:55.746790 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hrnxj" event={"ID":"00cbf4b6-5a38-46f2-9b62-7a5d4088bc9e","Type":"ContainerStarted","Data":"57e47f7dab3074e890498b450752fc69e386a56ba9485ffd9a0bd13712e98844"} Sep 08 11:37:57.668652 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:57.668609 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-756f55cbf4-kqcs8" podUID="9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b" containerName="console" containerID="cri-o://d1a692baff5d9f525f1feb7c17b3d91362fa31fee07640979f0ed594b7c8302e" gracePeriod=15 Sep 08 11:37:57.765401 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:57.765343 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hrnxj" event={"ID":"00cbf4b6-5a38-46f2-9b62-7a5d4088bc9e","Type":"ContainerStarted","Data":"3f9fc3ee030829f3f3a23e3c841b27f20d513513b22716d188fff50aca870f0f"} Sep 08 11:37:57.765547 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:57.765431 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hrnxj" Sep 08 11:37:57.790665 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:57.790615 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hrnxj" podStartSLOduration=2.177605324 podStartE2EDuration="3.790598336s" totalImagesPullingTime="1.612993012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:37:57.789316781 +0000 UTC m=+466.648504120" watchObservedRunningTime="2026-09-08 11:37:57.790598336 +0000 UTC m=+466.649785654" Sep 08 11:37:57.886821 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:57.886789 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-ffxcd"] Sep 08 11:37:57.890348 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:57.890333 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-ffxcd" Sep 08 11:37:57.894151 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:57.894131 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-82rqt\"" Sep 08 11:37:57.906067 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:57.906043 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-ffxcd"] Sep 08 11:37:57.923196 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:57.923143 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:37:57.929169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:57.929149 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sqgd\" (UniqueName: \"kubernetes.io/projected/70da0468-a13e-4146-a5be-0dd76443cd46-kube-api-access-6sqgd\") pod \"authorino-operator-68d88948d8-ffxcd\" (UID: \"70da0468-a13e-4146-a5be-0dd76443cd46\") " pod="kuadrant-system/authorino-operator-68d88948d8-ffxcd" Sep 08 11:37:58.030072 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.030042 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-trusted-ca-bundle\") pod \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " Sep 08 11:37:58.030244 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.030102 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-oauth-serving-cert\") pod \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " Sep 08 11:37:58.030244 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.030127 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ckrn\" (UniqueName: \"kubernetes.io/projected/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-kube-api-access-2ckrn\") pod \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " Sep 08 11:37:58.030244 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.030147 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-oauth-config\") pod \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " Sep 08 11:37:58.030244 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.030173 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-service-ca\") pod \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " Sep 08 11:37:58.030244 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.030211 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-config\") pod \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " Sep 08 11:37:58.030521 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.030298 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-serving-cert\") pod \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\" (UID: \"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b\") " Sep 08 11:37:58.030573 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.030552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sqgd\" (UniqueName: \"kubernetes.io/projected/70da0468-a13e-4146-a5be-0dd76443cd46-kube-api-access-6sqgd\") pod \"authorino-operator-68d88948d8-ffxcd\" (UID: \"70da0468-a13e-4146-a5be-0dd76443cd46\") " pod="kuadrant-system/authorino-operator-68d88948d8-ffxcd" Sep 08 11:37:58.030698 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.030553 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b" (UID: "9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:37:58.030750 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.030634 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b" (UID: "9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:37:58.030750 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.030651 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-config" (OuterVolumeSpecName: "console-config") pod "9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b" (UID: "9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:37:58.030750 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.030656 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-service-ca" (OuterVolumeSpecName: "service-ca") pod "9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b" (UID: "9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:37:58.032519 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.032409 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b" (UID: "9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:37:58.032519 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.032481 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b" (UID: "9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:37:58.032652 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.032579 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-kube-api-access-2ckrn" (OuterVolumeSpecName: "kube-api-access-2ckrn") pod "9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b" (UID: "9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b"). InnerVolumeSpecName "kube-api-access-2ckrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:37:58.039970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.039947 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sqgd\" (UniqueName: \"kubernetes.io/projected/70da0468-a13e-4146-a5be-0dd76443cd46-kube-api-access-6sqgd\") pod \"authorino-operator-68d88948d8-ffxcd\" (UID: \"70da0468-a13e-4146-a5be-0dd76443cd46\") " pod="kuadrant-system/authorino-operator-68d88948d8-ffxcd" Sep 08 11:37:58.131765 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.131739 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-trusted-ca-bundle\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:58.131765 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.131761 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-oauth-serving-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:58.131765 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.131770 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ckrn\" (UniqueName: \"kubernetes.io/projected/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-kube-api-access-2ckrn\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:58.131972 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.131780 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-oauth-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:58.131972 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.131789 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:58.131972 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.131797 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-config\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:58.131972 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.131806 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b-console-serving-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:37:58.217359 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.217300 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-ffxcd" Sep 08 11:37:58.498886 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.498862 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-ffxcd"] Sep 08 11:37:58.772652 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.772558 2564 generic.go:356] "Generic (PLEG): container finished" podID="9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b" containerID="d1a692baff5d9f525f1feb7c17b3d91362fa31fee07640979f0ed594b7c8302e" exitCode=0 Sep 08 11:37:58.772652 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.772615 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-756f55cbf4-kqcs8" Sep 08 11:37:58.772652 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.772640 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-756f55cbf4-kqcs8" event={"ID":"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b","Type":"ContainerDied","Data":"d1a692baff5d9f525f1feb7c17b3d91362fa31fee07640979f0ed594b7c8302e"} Sep 08 11:37:58.773188 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.772691 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-756f55cbf4-kqcs8" event={"ID":"9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b","Type":"ContainerDied","Data":"eacc463bef8c77df2a9c7632b3dfe4972606ae5ce9f1c6931857da194364f27a"} Sep 08 11:37:58.773188 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.772716 2564 scope.go:117] "RemoveContainer" containerID="d1a692baff5d9f525f1feb7c17b3d91362fa31fee07640979f0ed594b7c8302e" Sep 08 11:37:58.799002 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.798970 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-756f55cbf4-kqcs8"] Sep 08 11:37:58.803710 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:58.803690 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-756f55cbf4-kqcs8"] Sep 08 11:37:59.621152 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:37:59.621111 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70da0468_a13e_4146_a5be_0dd76443cd46.slice/crio-3e88770c35c7121ca44d822051a37f6e4f5e33a229c8ce510d47339b1c36b0f2 WatchSource:0}: Error finding container 3e88770c35c7121ca44d822051a37f6e4f5e33a229c8ce510d47339b1c36b0f2: Status 404 returned error can't find the container with id 3e88770c35c7121ca44d822051a37f6e4f5e33a229c8ce510d47339b1c36b0f2 Sep 08 11:37:59.624133 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:59.624112 2564 scope.go:117] "RemoveContainer" containerID="d1a692baff5d9f525f1feb7c17b3d91362fa31fee07640979f0ed594b7c8302e" Sep 08 11:37:59.624427 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:37:59.624405 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a692baff5d9f525f1feb7c17b3d91362fa31fee07640979f0ed594b7c8302e\": container with ID starting with d1a692baff5d9f525f1feb7c17b3d91362fa31fee07640979f0ed594b7c8302e not found: ID does not exist" containerID="d1a692baff5d9f525f1feb7c17b3d91362fa31fee07640979f0ed594b7c8302e" Sep 08 11:37:59.624522 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:59.624441 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a692baff5d9f525f1feb7c17b3d91362fa31fee07640979f0ed594b7c8302e"} err="failed to get container status \"d1a692baff5d9f525f1feb7c17b3d91362fa31fee07640979f0ed594b7c8302e\": rpc error: code = NotFound desc = could not find container \"d1a692baff5d9f525f1feb7c17b3d91362fa31fee07640979f0ed594b7c8302e\": container with ID starting with d1a692baff5d9f525f1feb7c17b3d91362fa31fee07640979f0ed594b7c8302e not found: ID does not exist" Sep 08 11:37:59.653050 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:59.653020 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b" path="/var/lib/kubelet/pods/9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b/volumes" Sep 08 11:37:59.780822 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:59.780791 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" event={"ID":"aad12b53-e462-450b-a782-e2a047bae7a6","Type":"ContainerStarted","Data":"bf22d97a368da1acd89f894223fb8de8befdf6510e7548f5c6c76eef73962cc0"} Sep 08 11:37:59.781218 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:59.780897 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" Sep 08 11:37:59.782478 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:59.782450 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-ffxcd" event={"ID":"70da0468-a13e-4146-a5be-0dd76443cd46","Type":"ContainerStarted","Data":"3e88770c35c7121ca44d822051a37f6e4f5e33a229c8ce510d47339b1c36b0f2"} Sep 08 11:37:59.815176 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:37:59.815132 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" podStartSLOduration=1.4578826999999999 podStartE2EDuration="9.815113148s" totalImagesPullingTime="8.357230448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:37:59.812839492 +0000 UTC m=+468.672026834" watchObservedRunningTime="2026-09-08 11:37:59.815113148 +0000 UTC m=+468.674300466" Sep 08 11:38:01.795537 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:01.795501 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-ffxcd" event={"ID":"70da0468-a13e-4146-a5be-0dd76443cd46","Type":"ContainerStarted","Data":"20a41e5ef9d80ae1f23e0335874558a4119ac56f015f571ce5a11985234d56ee"} Sep 08 11:38:01.795880 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:01.795591 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-ffxcd" Sep 08 11:38:01.816637 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:01.816551 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-ffxcd" podStartSLOduration=3.060700841 podStartE2EDuration="4.816531292s" totalImagesPullingTime="1.755830451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:38:01.815318092 +0000 UTC m=+470.674505411" watchObservedRunningTime="2026-09-08 11:38:01.816531292 +0000 UTC m=+470.675718611" Sep 08 11:38:08.775519 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:08.775487 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-hrnxj" Sep 08 11:38:10.791027 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:10.790989 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" Sep 08 11:38:12.803531 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:12.803497 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-ffxcd" Sep 08 11:38:12.919807 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:12.919778 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v"] Sep 08 11:38:12.919975 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:12.919956 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" podUID="aad12b53-e462-450b-a782-e2a047bae7a6" containerName="manager" containerID="cri-o://bf22d97a368da1acd89f894223fb8de8befdf6510e7548f5c6c76eef73962cc0" gracePeriod=2 Sep 08 11:38:12.931924 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:12.931895 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v"] Sep 08 11:38:12.933094 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:38:12.933058 2564 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"aad12b53-e462-450b-a782-e2a047bae7a6\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-58fc56757c-tjw8v\": pods \"kuadrant-operator-controller-manager-58fc56757c-tjw8v\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" Sep 08 11:38:12.946144 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:12.946113 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7"] Sep 08 11:38:12.946460 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:12.946446 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad12b53-e462-450b-a782-e2a047bae7a6" containerName="manager" Sep 08 11:38:12.946509 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:12.946462 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad12b53-e462-450b-a782-e2a047bae7a6" containerName="manager" Sep 08 11:38:12.946509 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:12.946489 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b" containerName="console" Sep 08 11:38:12.946509 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:12.946494 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b" containerName="console" Sep 08 11:38:12.946599 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:12.946548 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d67e861-b4e2-4ac8-84e4-4ec9ca82a93b" containerName="console" Sep 08 11:38:12.946599 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:12.946556 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad12b53-e462-450b-a782-e2a047bae7a6" containerName="manager" Sep 08 11:38:12.949623 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:12.949609 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" Sep 08 11:38:12.965761 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:12.965734 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7"] Sep 08 11:38:12.981731 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:38:12.981697 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-tjw8v\" is forbidden: User \"system:node:ip-10-0-133-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-187.ec2.internal' and this object" podUID="aad12b53-e462-450b-a782-e2a047bae7a6" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" Sep 08 11:38:13.058226 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.058148 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bf85a7e-7280-4e5b-9224-efebf8118480-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-bhcb7\" (UID: \"5bf85a7e-7280-4e5b-9224-efebf8118480\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" Sep 08 11:38:13.058337 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.058294 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj5ph\" (UniqueName: \"kubernetes.io/projected/5bf85a7e-7280-4e5b-9224-efebf8118480-kube-api-access-mj5ph\") pod \"kuadrant-operator-controller-manager-58fc56757c-bhcb7\" (UID: \"5bf85a7e-7280-4e5b-9224-efebf8118480\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" Sep 08 11:38:13.159396 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.159351 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bf85a7e-7280-4e5b-9224-efebf8118480-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-bhcb7\" (UID: \"5bf85a7e-7280-4e5b-9224-efebf8118480\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" Sep 08 11:38:13.159519 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.159443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj5ph\" (UniqueName: \"kubernetes.io/projected/5bf85a7e-7280-4e5b-9224-efebf8118480-kube-api-access-mj5ph\") pod \"kuadrant-operator-controller-manager-58fc56757c-bhcb7\" (UID: \"5bf85a7e-7280-4e5b-9224-efebf8118480\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" Sep 08 11:38:13.159561 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.159541 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" Sep 08 11:38:13.159712 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.159692 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bf85a7e-7280-4e5b-9224-efebf8118480-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-bhcb7\" (UID: \"5bf85a7e-7280-4e5b-9224-efebf8118480\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" Sep 08 11:38:13.163634 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:38:13.163609 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-tjw8v\" is forbidden: User \"system:node:ip-10-0-133-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-187.ec2.internal' and this object" podUID="aad12b53-e462-450b-a782-e2a047bae7a6" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" Sep 08 11:38:13.181995 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.181973 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj5ph\" (UniqueName: \"kubernetes.io/projected/5bf85a7e-7280-4e5b-9224-efebf8118480-kube-api-access-mj5ph\") pod \"kuadrant-operator-controller-manager-58fc56757c-bhcb7\" (UID: \"5bf85a7e-7280-4e5b-9224-efebf8118480\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" Sep 08 11:38:13.260662 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.260631 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfwdc\" (UniqueName: \"kubernetes.io/projected/aad12b53-e462-450b-a782-e2a047bae7a6-kube-api-access-xfwdc\") pod \"aad12b53-e462-450b-a782-e2a047bae7a6\" (UID: \"aad12b53-e462-450b-a782-e2a047bae7a6\") " Sep 08 11:38:13.260786 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.260681 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aad12b53-e462-450b-a782-e2a047bae7a6-extensions-socket-volume\") pod \"aad12b53-e462-450b-a782-e2a047bae7a6\" (UID: \"aad12b53-e462-450b-a782-e2a047bae7a6\") " Sep 08 11:38:13.261051 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.261030 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad12b53-e462-450b-a782-e2a047bae7a6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "aad12b53-e462-450b-a782-e2a047bae7a6" (UID: "aad12b53-e462-450b-a782-e2a047bae7a6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:38:13.262838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.262809 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad12b53-e462-450b-a782-e2a047bae7a6-kube-api-access-xfwdc" (OuterVolumeSpecName: "kube-api-access-xfwdc") pod "aad12b53-e462-450b-a782-e2a047bae7a6" (UID: "aad12b53-e462-450b-a782-e2a047bae7a6"). InnerVolumeSpecName "kube-api-access-xfwdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:38:13.298993 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.298965 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" Sep 08 11:38:13.362753 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.362069 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfwdc\" (UniqueName: \"kubernetes.io/projected/aad12b53-e462-450b-a782-e2a047bae7a6-kube-api-access-xfwdc\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:38:13.362753 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.362103 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aad12b53-e462-450b-a782-e2a047bae7a6-extensions-socket-volume\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:38:13.601503 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.601479 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7"] Sep 08 11:38:13.602152 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:38:13.602127 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf85a7e_7280_4e5b_9224_efebf8118480.slice/crio-73603420bed76364acc609b759840cf29de01027007bb98e6c0430e54df9f378 WatchSource:0}: Error finding container 73603420bed76364acc609b759840cf29de01027007bb98e6c0430e54df9f378: Status 404 returned error can't find the container with id 73603420bed76364acc609b759840cf29de01027007bb98e6c0430e54df9f378 Sep 08 11:38:13.653272 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.653248 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aad12b53-e462-450b-a782-e2a047bae7a6" path="/var/lib/kubelet/pods/aad12b53-e462-450b-a782-e2a047bae7a6/volumes" Sep 08 11:38:13.861103 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.861028 2564 generic.go:356] "Generic (PLEG): container finished" podID="aad12b53-e462-450b-a782-e2a047bae7a6" containerID="bf22d97a368da1acd89f894223fb8de8befdf6510e7548f5c6c76eef73962cc0" exitCode=0 Sep 08 11:38:13.861103 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.861070 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tjw8v" Sep 08 11:38:13.861597 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.861123 2564 scope.go:117] "RemoveContainer" containerID="bf22d97a368da1acd89f894223fb8de8befdf6510e7548f5c6c76eef73962cc0" Sep 08 11:38:13.863419 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.863313 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" event={"ID":"5bf85a7e-7280-4e5b-9224-efebf8118480","Type":"ContainerStarted","Data":"0f2ae122345b35b5d24c9e41b418d70ca68ed13bfa0a8f03c5caf86438db3edf"} Sep 08 11:38:13.863419 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.863391 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" event={"ID":"5bf85a7e-7280-4e5b-9224-efebf8118480","Type":"ContainerStarted","Data":"73603420bed76364acc609b759840cf29de01027007bb98e6c0430e54df9f378"} Sep 08 11:38:13.863570 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.863425 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" Sep 08 11:38:13.869526 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.869509 2564 scope.go:117] "RemoveContainer" containerID="bf22d97a368da1acd89f894223fb8de8befdf6510e7548f5c6c76eef73962cc0" Sep 08 11:38:13.869778 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:38:13.869759 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf22d97a368da1acd89f894223fb8de8befdf6510e7548f5c6c76eef73962cc0\": container with ID starting with bf22d97a368da1acd89f894223fb8de8befdf6510e7548f5c6c76eef73962cc0 not found: ID does not exist" containerID="bf22d97a368da1acd89f894223fb8de8befdf6510e7548f5c6c76eef73962cc0" Sep 08 11:38:13.869840 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.869785 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf22d97a368da1acd89f894223fb8de8befdf6510e7548f5c6c76eef73962cc0"} err="failed to get container status \"bf22d97a368da1acd89f894223fb8de8befdf6510e7548f5c6c76eef73962cc0\": rpc error: code = NotFound desc = could not find container \"bf22d97a368da1acd89f894223fb8de8befdf6510e7548f5c6c76eef73962cc0\": container with ID starting with bf22d97a368da1acd89f894223fb8de8befdf6510e7548f5c6c76eef73962cc0 not found: ID does not exist" Sep 08 11:38:13.887621 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:13.887579 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" podStartSLOduration=1.887566305 podStartE2EDuration="1.887566305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:38:13.885555151 +0000 UTC m=+482.744742491" watchObservedRunningTime="2026-09-08 11:38:13.887566305 +0000 UTC m=+482.746753623" Sep 08 11:38:24.871753 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:24.871722 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" Sep 08 11:38:31.015864 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.015828 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7"] Sep 08 11:38:31.016254 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.016077 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" podUID="5bf85a7e-7280-4e5b-9224-efebf8118480" containerName="manager" containerID="cri-o://0f2ae122345b35b5d24c9e41b418d70ca68ed13bfa0a8f03c5caf86438db3edf" gracePeriod=10 Sep 08 11:38:31.255065 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.255036 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c"] Sep 08 11:38:31.257763 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.257742 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" Sep 08 11:38:31.270157 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.270098 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" Sep 08 11:38:31.275223 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.275200 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c"] Sep 08 11:38:31.417470 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.417437 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj5ph\" (UniqueName: \"kubernetes.io/projected/5bf85a7e-7280-4e5b-9224-efebf8118480-kube-api-access-mj5ph\") pod \"5bf85a7e-7280-4e5b-9224-efebf8118480\" (UID: \"5bf85a7e-7280-4e5b-9224-efebf8118480\") " Sep 08 11:38:31.417647 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.417486 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bf85a7e-7280-4e5b-9224-efebf8118480-extensions-socket-volume\") pod \"5bf85a7e-7280-4e5b-9224-efebf8118480\" (UID: \"5bf85a7e-7280-4e5b-9224-efebf8118480\") " Sep 08 11:38:31.417647 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.417576 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/178c18c2-ad8f-433a-bf00-281d77b10d43-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-knz2c\" (UID: \"178c18c2-ad8f-433a-bf00-281d77b10d43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" Sep 08 11:38:31.417647 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.417634 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmhdp\" (UniqueName: \"kubernetes.io/projected/178c18c2-ad8f-433a-bf00-281d77b10d43-kube-api-access-cmhdp\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-knz2c\" (UID: \"178c18c2-ad8f-433a-bf00-281d77b10d43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" Sep 08 11:38:31.417932 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.417908 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf85a7e-7280-4e5b-9224-efebf8118480-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5bf85a7e-7280-4e5b-9224-efebf8118480" (UID: "5bf85a7e-7280-4e5b-9224-efebf8118480"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:38:31.419527 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.419505 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf85a7e-7280-4e5b-9224-efebf8118480-kube-api-access-mj5ph" (OuterVolumeSpecName: "kube-api-access-mj5ph") pod "5bf85a7e-7280-4e5b-9224-efebf8118480" (UID: "5bf85a7e-7280-4e5b-9224-efebf8118480"). InnerVolumeSpecName "kube-api-access-mj5ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:38:31.519126 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.519090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/178c18c2-ad8f-433a-bf00-281d77b10d43-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-knz2c\" (UID: \"178c18c2-ad8f-433a-bf00-281d77b10d43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" Sep 08 11:38:31.519292 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.519151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmhdp\" (UniqueName: \"kubernetes.io/projected/178c18c2-ad8f-433a-bf00-281d77b10d43-kube-api-access-cmhdp\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-knz2c\" (UID: \"178c18c2-ad8f-433a-bf00-281d77b10d43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" Sep 08 11:38:31.519292 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.519192 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj5ph\" (UniqueName: \"kubernetes.io/projected/5bf85a7e-7280-4e5b-9224-efebf8118480-kube-api-access-mj5ph\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:38:31.519292 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.519201 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bf85a7e-7280-4e5b-9224-efebf8118480-extensions-socket-volume\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:38:31.519483 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.519462 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/178c18c2-ad8f-433a-bf00-281d77b10d43-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-knz2c\" (UID: \"178c18c2-ad8f-433a-bf00-281d77b10d43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" Sep 08 11:38:31.532960 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.532893 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmhdp\" (UniqueName: \"kubernetes.io/projected/178c18c2-ad8f-433a-bf00-281d77b10d43-kube-api-access-cmhdp\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-knz2c\" (UID: \"178c18c2-ad8f-433a-bf00-281d77b10d43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" Sep 08 11:38:31.578229 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.578201 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" Sep 08 11:38:31.859806 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.859752 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c"] Sep 08 11:38:31.860448 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:38:31.860425 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178c18c2_ad8f_433a_bf00_281d77b10d43.slice/crio-f4384d26e4c88f744f0e8b2b1f423a6bb8590709df80544bc9759d37a5ed35d7 WatchSource:0}: Error finding container f4384d26e4c88f744f0e8b2b1f423a6bb8590709df80544bc9759d37a5ed35d7: Status 404 returned error can't find the container with id f4384d26e4c88f744f0e8b2b1f423a6bb8590709df80544bc9759d37a5ed35d7 Sep 08 11:38:31.964247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.964213 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" event={"ID":"178c18c2-ad8f-433a-bf00-281d77b10d43","Type":"ContainerStarted","Data":"0f8d56fafa64bd211c41296bf067217c36e1beeeb4a48e3b5a24278f34c999b5"} Sep 08 11:38:31.964422 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.964273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" event={"ID":"178c18c2-ad8f-433a-bf00-281d77b10d43","Type":"ContainerStarted","Data":"f4384d26e4c88f744f0e8b2b1f423a6bb8590709df80544bc9759d37a5ed35d7"} Sep 08 11:38:31.964422 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.964399 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" Sep 08 11:38:31.965584 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.965561 2564 generic.go:356] "Generic (PLEG): container finished" podID="5bf85a7e-7280-4e5b-9224-efebf8118480" containerID="0f2ae122345b35b5d24c9e41b418d70ca68ed13bfa0a8f03c5caf86438db3edf" exitCode=0 Sep 08 11:38:31.965643 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.965595 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" Sep 08 11:38:31.965643 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.965610 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" event={"ID":"5bf85a7e-7280-4e5b-9224-efebf8118480","Type":"ContainerDied","Data":"0f2ae122345b35b5d24c9e41b418d70ca68ed13bfa0a8f03c5caf86438db3edf"} Sep 08 11:38:31.965721 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.965645 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7" event={"ID":"5bf85a7e-7280-4e5b-9224-efebf8118480","Type":"ContainerDied","Data":"73603420bed76364acc609b759840cf29de01027007bb98e6c0430e54df9f378"} Sep 08 11:38:31.965721 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.965672 2564 scope.go:117] "RemoveContainer" containerID="0f2ae122345b35b5d24c9e41b418d70ca68ed13bfa0a8f03c5caf86438db3edf" Sep 08 11:38:31.974142 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.974128 2564 scope.go:117] "RemoveContainer" containerID="0f2ae122345b35b5d24c9e41b418d70ca68ed13bfa0a8f03c5caf86438db3edf" Sep 08 11:38:31.974537 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:38:31.974517 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2ae122345b35b5d24c9e41b418d70ca68ed13bfa0a8f03c5caf86438db3edf\": container with ID starting with 0f2ae122345b35b5d24c9e41b418d70ca68ed13bfa0a8f03c5caf86438db3edf not found: ID does not exist" containerID="0f2ae122345b35b5d24c9e41b418d70ca68ed13bfa0a8f03c5caf86438db3edf" Sep 08 11:38:31.974607 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:31.974547 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2ae122345b35b5d24c9e41b418d70ca68ed13bfa0a8f03c5caf86438db3edf"} err="failed to get container status \"0f2ae122345b35b5d24c9e41b418d70ca68ed13bfa0a8f03c5caf86438db3edf\": rpc error: code = NotFound desc = could not find container \"0f2ae122345b35b5d24c9e41b418d70ca68ed13bfa0a8f03c5caf86438db3edf\": container with ID starting with 0f2ae122345b35b5d24c9e41b418d70ca68ed13bfa0a8f03c5caf86438db3edf not found: ID does not exist" Sep 08 11:38:32.003999 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:32.003950 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" podStartSLOduration=1.003936331 podStartE2EDuration="1.003936331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:38:32.002807957 +0000 UTC m=+500.861995275" watchObservedRunningTime="2026-09-08 11:38:32.003936331 +0000 UTC m=+500.863123649" Sep 08 11:38:32.029947 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:32.029918 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7"] Sep 08 11:38:32.036101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:32.036078 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bhcb7"] Sep 08 11:38:33.655395 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:33.655336 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bf85a7e-7280-4e5b-9224-efebf8118480" path="/var/lib/kubelet/pods/5bf85a7e-7280-4e5b-9224-efebf8118480/volumes" Sep 08 11:38:42.980078 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:42.980050 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" Sep 08 11:38:50.538648 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.538612 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:38:50.539043 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.539019 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf85a7e-7280-4e5b-9224-efebf8118480" containerName="manager" Sep 08 11:38:50.539043 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.539037 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf85a7e-7280-4e5b-9224-efebf8118480" containerName="manager" Sep 08 11:38:50.539169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.539154 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf85a7e-7280-4e5b-9224-efebf8118480" containerName="manager" Sep 08 11:38:50.542478 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.542453 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg" Sep 08 11:38:50.545069 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.545047 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 08 11:38:50.545197 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.545104 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vjdc9\"" Sep 08 11:38:50.554687 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.554661 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:38:50.560152 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.560125 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvzqj\" (UniqueName: \"kubernetes.io/projected/b7c28493-f1fd-49bc-a64a-cef03d75acf9-kube-api-access-fvzqj\") pod \"limitador-limitador-85d9db4d5d-nn5kg\" (UID: \"b7c28493-f1fd-49bc-a64a-cef03d75acf9\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg" Sep 08 11:38:50.560350 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.560174 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7c28493-f1fd-49bc-a64a-cef03d75acf9-config-file\") pod \"limitador-limitador-85d9db4d5d-nn5kg\" (UID: \"b7c28493-f1fd-49bc-a64a-cef03d75acf9\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg" Sep 08 11:38:50.583580 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.583549 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:38:50.661629 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.661590 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvzqj\" (UniqueName: \"kubernetes.io/projected/b7c28493-f1fd-49bc-a64a-cef03d75acf9-kube-api-access-fvzqj\") pod \"limitador-limitador-85d9db4d5d-nn5kg\" (UID: \"b7c28493-f1fd-49bc-a64a-cef03d75acf9\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg" Sep 08 11:38:50.661802 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.661706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7c28493-f1fd-49bc-a64a-cef03d75acf9-config-file\") pod \"limitador-limitador-85d9db4d5d-nn5kg\" (UID: \"b7c28493-f1fd-49bc-a64a-cef03d75acf9\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg" Sep 08 11:38:50.662440 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.662418 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7c28493-f1fd-49bc-a64a-cef03d75acf9-config-file\") pod \"limitador-limitador-85d9db4d5d-nn5kg\" (UID: \"b7c28493-f1fd-49bc-a64a-cef03d75acf9\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg" Sep 08 11:38:50.671479 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.671459 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvzqj\" (UniqueName: \"kubernetes.io/projected/b7c28493-f1fd-49bc-a64a-cef03d75acf9-kube-api-access-fvzqj\") pod \"limitador-limitador-85d9db4d5d-nn5kg\" (UID: \"b7c28493-f1fd-49bc-a64a-cef03d75acf9\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg" Sep 08 11:38:50.853756 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:50.853664 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg" Sep 08 11:38:51.126919 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:51.126895 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:38:51.128574 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:38:51.128548 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c28493_f1fd_49bc_a64a_cef03d75acf9.slice/crio-519eef6d32f707cf7494b4e5200c4feefc5eb5c7796d65a09e38ce57b05b652a WatchSource:0}: Error finding container 519eef6d32f707cf7494b4e5200c4feefc5eb5c7796d65a09e38ce57b05b652a: Status 404 returned error can't find the container with id 519eef6d32f707cf7494b4e5200c4feefc5eb5c7796d65a09e38ce57b05b652a Sep 08 11:38:52.085894 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:52.085859 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg" event={"ID":"b7c28493-f1fd-49bc-a64a-cef03d75acf9","Type":"ContainerStarted","Data":"519eef6d32f707cf7494b4e5200c4feefc5eb5c7796d65a09e38ce57b05b652a"} Sep 08 11:38:54.097493 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:54.097465 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg" event={"ID":"b7c28493-f1fd-49bc-a64a-cef03d75acf9","Type":"ContainerStarted","Data":"7d7a4fb76dbd680ecfced2805b20ae63cbaf7555a5fcf5c52ec6a9cd5f8b0208"} Sep 08 11:38:54.097819 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:54.097579 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg" Sep 08 11:38:54.120192 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:38:54.120144 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg" podStartSLOduration=1.25764998 podStartE2EDuration="4.12013047s" totalImagesPullingTime="2.86248049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:38:54.118433047 +0000 UTC m=+522.977620402" watchObservedRunningTime="2026-09-08 11:38:54.12013047 +0000 UTC m=+522.979317860" Sep 08 11:39:05.104494 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:39:05.104409 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg" Sep 08 11:40:11.566300 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:40:11.566270 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 11:40:11.567406 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:40:11.567363 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 11:40:20.691662 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:40:20.691610 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:40:40.049263 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:40:40.049178 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:40:42.943067 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:40:42.943035 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:40:55.044926 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:40:55.044890 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:41:05.730178 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.730139 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-7v499"] Sep 08 11:41:05.733830 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.733801 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.737724 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.737698 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 08 11:41:05.737857 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.737730 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 08 11:41:05.737857 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.737751 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 08 11:41:05.737857 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.737732 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pkvgk\"" Sep 08 11:41:05.743306 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.743286 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-7v499"] Sep 08 11:41:05.779042 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.779018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5752df83-029c-4b2f-bc14-57c2e46646e3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.779177 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.779080 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knql9\" (UniqueName: \"kubernetes.io/projected/5752df83-029c-4b2f-bc14-57c2e46646e3-kube-api-access-knql9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.779177 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.779162 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5752df83-029c-4b2f-bc14-57c2e46646e3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.779292 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.779198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5752df83-029c-4b2f-bc14-57c2e46646e3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.779292 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.779222 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5752df83-029c-4b2f-bc14-57c2e46646e3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.779362 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.779295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5752df83-029c-4b2f-bc14-57c2e46646e3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.880086 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.880055 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knql9\" (UniqueName: \"kubernetes.io/projected/5752df83-029c-4b2f-bc14-57c2e46646e3-kube-api-access-knql9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.880286 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.880095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5752df83-029c-4b2f-bc14-57c2e46646e3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.880286 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.880113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5752df83-029c-4b2f-bc14-57c2e46646e3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.880286 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.880127 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5752df83-029c-4b2f-bc14-57c2e46646e3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.880286 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.880155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5752df83-029c-4b2f-bc14-57c2e46646e3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.880286 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.880221 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5752df83-029c-4b2f-bc14-57c2e46646e3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.880616 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.880590 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5752df83-029c-4b2f-bc14-57c2e46646e3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.880668 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.880630 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5752df83-029c-4b2f-bc14-57c2e46646e3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.880668 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.880655 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5752df83-029c-4b2f-bc14-57c2e46646e3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.882460 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.882442 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5752df83-029c-4b2f-bc14-57c2e46646e3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.882862 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.882841 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5752df83-029c-4b2f-bc14-57c2e46646e3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:05.889233 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:05.889212 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knql9\" (UniqueName: \"kubernetes.io/projected/5752df83-029c-4b2f-bc14-57c2e46646e3-kube-api-access-knql9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7v499\" (UID: \"5752df83-029c-4b2f-bc14-57c2e46646e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:06.046026 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:06.045936 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:06.326023 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:06.325989 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-7v499"] Sep 08 11:41:06.327272 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:41:06.327247 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5752df83_029c_4b2f_bc14_57c2e46646e3.slice/crio-136342bb212b3751245c652c69a3ea2f1b62002e50319e3bb8368186ec59692d WatchSource:0}: Error finding container 136342bb212b3751245c652c69a3ea2f1b62002e50319e3bb8368186ec59692d: Status 404 returned error can't find the container with id 136342bb212b3751245c652c69a3ea2f1b62002e50319e3bb8368186ec59692d Sep 08 11:41:06.328749 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:06.328729 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 11:41:06.811169 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:06.811122 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" event={"ID":"5752df83-029c-4b2f-bc14-57c2e46646e3","Type":"ContainerStarted","Data":"136342bb212b3751245c652c69a3ea2f1b62002e50319e3bb8368186ec59692d"} Sep 08 11:41:07.136301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:07.136273 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:41:07.817839 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:07.817797 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" event={"ID":"5752df83-029c-4b2f-bc14-57c2e46646e3","Type":"ContainerStarted","Data":"e71702c131c1fae3cdf01a6a94f05e7fc096c30a1b286f6fe59cdd186c1be828"} Sep 08 11:41:07.837046 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:07.837001 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" podStartSLOduration=1.871991418 podStartE2EDuration="2.836986269s" totalImagesPullingTime="964.994851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:41:07.835812701 +0000 UTC m=+656.695000038" watchObservedRunningTime="2026-09-08 11:41:07.836986269 +0000 UTC m=+656.696173615" Sep 08 11:41:12.022752 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.022655 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t"] Sep 08 11:41:12.026322 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.026301 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.028973 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.028950 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 08 11:41:12.038138 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.038115 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t"] Sep 08 11:41:12.132888 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.132857 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqn7t\" (UniqueName: \"kubernetes.io/projected/73a2e868-7b61-4f6e-81b4-de979eb45479-kube-api-access-cqn7t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.133058 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.132897 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a2e868-7b61-4f6e-81b4-de979eb45479-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.133058 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.132934 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a2e868-7b61-4f6e-81b4-de979eb45479-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.133058 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.133002 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a2e868-7b61-4f6e-81b4-de979eb45479-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.133058 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.133036 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a2e868-7b61-4f6e-81b4-de979eb45479-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.133058 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.133050 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a2e868-7b61-4f6e-81b4-de979eb45479-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.234484 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.234451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqn7t\" (UniqueName: \"kubernetes.io/projected/73a2e868-7b61-4f6e-81b4-de979eb45479-kube-api-access-cqn7t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.234669 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.234492 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a2e868-7b61-4f6e-81b4-de979eb45479-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.234669 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.234528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a2e868-7b61-4f6e-81b4-de979eb45479-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.234669 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.234587 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a2e868-7b61-4f6e-81b4-de979eb45479-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.234669 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.234616 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a2e868-7b61-4f6e-81b4-de979eb45479-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.234669 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.234640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a2e868-7b61-4f6e-81b4-de979eb45479-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.234924 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.234858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73a2e868-7b61-4f6e-81b4-de979eb45479-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.234967 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.234931 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73a2e868-7b61-4f6e-81b4-de979eb45479-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.235058 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.235042 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73a2e868-7b61-4f6e-81b4-de979eb45479-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.236885 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.236863 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73a2e868-7b61-4f6e-81b4-de979eb45479-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.237140 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.237122 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a2e868-7b61-4f6e-81b4-de979eb45479-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.244353 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.244325 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqn7t\" (UniqueName: \"kubernetes.io/projected/73a2e868-7b61-4f6e-81b4-de979eb45479-kube-api-access-cqn7t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t\" (UID: \"73a2e868-7b61-4f6e-81b4-de979eb45479\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.337569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.337489 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:12.617439 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.617413 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t"] Sep 08 11:41:12.618435 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:41:12.618407 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a2e868_7b61_4f6e_81b4_de979eb45479.slice/crio-5b4b31b62744e2089da177c2e7d1a92a510af22d2de9fc47db32dda056c7a002 WatchSource:0}: Error finding container 5b4b31b62744e2089da177c2e7d1a92a510af22d2de9fc47db32dda056c7a002: Status 404 returned error can't find the container with id 5b4b31b62744e2089da177c2e7d1a92a510af22d2de9fc47db32dda056c7a002 Sep 08 11:41:12.847417 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:12.847389 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" event={"ID":"73a2e868-7b61-4f6e-81b4-de979eb45479","Type":"ContainerStarted","Data":"5b4b31b62744e2089da177c2e7d1a92a510af22d2de9fc47db32dda056c7a002"} Sep 08 11:41:13.233926 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:13.233894 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:41:13.854851 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:13.854812 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" event={"ID":"73a2e868-7b61-4f6e-81b4-de979eb45479","Type":"ContainerStarted","Data":"1db3fb17c5b12955528256eb00bd0f879d4540efe58d14275b03a9be6d131f16"} Sep 08 11:41:13.875674 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:13.875617 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" podStartSLOduration=1.6651638050000002 podStartE2EDuration="1.875598245s" totalImagesPullingTime="210.43444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:41:13.874241741 +0000 UTC m=+662.733429060" watchObservedRunningTime="2026-09-08 11:41:13.875598245 +0000 UTC m=+662.734785564" Sep 08 11:41:16.046390 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:16.046342 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:16.046775 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:16.046420 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:16.059031 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:16.059004 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:16.887153 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:16.887114 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7v499" Sep 08 11:41:22.338334 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:22.338302 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:22.338800 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:22.338407 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:22.350839 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:22.350813 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:22.918096 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:22.918065 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t" Sep 08 11:41:39.441623 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:39.441586 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:41:46.148916 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:41:46.148882 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:42:54.747827 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:42:54.747791 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:42:56.384640 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:42:56.384605 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7449-140146733610160"] Sep 08 11:42:56.388051 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:42:56.388030 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7449-140146733610160" Sep 08 11:42:56.390849 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:42:56.390826 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zbjsb\"" Sep 08 11:42:56.396356 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:42:56.396330 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7449-140146733610160"] Sep 08 11:42:56.448076 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:42:56.448043 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnjbp\" (UniqueName: \"kubernetes.io/projected/6702664a-2622-4663-921a-bbbecef96814-kube-api-access-gnjbp\") pod \"test-curl-7449-140146733610160\" (UID: \"6702664a-2622-4663-921a-bbbecef96814\") " pod="opendatahub/test-curl-7449-140146733610160" Sep 08 11:42:56.548655 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:42:56.548617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnjbp\" (UniqueName: \"kubernetes.io/projected/6702664a-2622-4663-921a-bbbecef96814-kube-api-access-gnjbp\") pod \"test-curl-7449-140146733610160\" (UID: \"6702664a-2622-4663-921a-bbbecef96814\") " pod="opendatahub/test-curl-7449-140146733610160" Sep 08 11:42:56.559889 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:42:56.559858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnjbp\" (UniqueName: \"kubernetes.io/projected/6702664a-2622-4663-921a-bbbecef96814-kube-api-access-gnjbp\") pod \"test-curl-7449-140146733610160\" (UID: \"6702664a-2622-4663-921a-bbbecef96814\") " pod="opendatahub/test-curl-7449-140146733610160" Sep 08 11:42:56.697746 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:42:56.697668 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7449-140146733610160" Sep 08 11:42:56.976433 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:42:56.976405 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7449-140146733610160"] Sep 08 11:42:56.977754 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:42:56.977724 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6702664a_2622_4663_921a_bbbecef96814.slice/crio-eecf9677b571e59a07b8d448ca09c4f0c24e490196e632eda617ce5e04fdba31 WatchSource:0}: Error finding container eecf9677b571e59a07b8d448ca09c4f0c24e490196e632eda617ce5e04fdba31: Status 404 returned error can't find the container with id eecf9677b571e59a07b8d448ca09c4f0c24e490196e632eda617ce5e04fdba31 Sep 08 11:42:57.423895 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:42:57.423861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7449-140146733610160" event={"ID":"6702664a-2622-4663-921a-bbbecef96814","Type":"ContainerStarted","Data":"eecf9677b571e59a07b8d448ca09c4f0c24e490196e632eda617ce5e04fdba31"} Sep 08 11:42:59.438621 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:42:59.438587 2564 generic.go:356] "Generic (PLEG): container finished" podID="6702664a-2622-4663-921a-bbbecef96814" containerID="6e190ca4dcd385947bd03f683cefcac3f7213582b1e59186a6b994a1a42f6d01" exitCode=0 Sep 08 11:42:59.439006 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:42:59.438678 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7449-140146733610160" event={"ID":"6702664a-2622-4663-921a-bbbecef96814","Type":"ContainerDied","Data":"6e190ca4dcd385947bd03f683cefcac3f7213582b1e59186a6b994a1a42f6d01"} Sep 08 11:43:00.574346 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:00.574320 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7449-140146733610160" Sep 08 11:43:00.611949 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:00.611920 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7449-140146733610160_6702664a-2622-4663-921a-bbbecef96814/test-curl-7449-140146733610160/0.log" Sep 08 11:43:00.637727 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:00.637696 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7449-140146733610160"] Sep 08 11:43:00.643831 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:00.643807 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7449-140146733610160"] Sep 08 11:43:00.685153 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:00.685131 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnjbp\" (UniqueName: \"kubernetes.io/projected/6702664a-2622-4663-921a-bbbecef96814-kube-api-access-gnjbp\") pod \"6702664a-2622-4663-921a-bbbecef96814\" (UID: \"6702664a-2622-4663-921a-bbbecef96814\") " Sep 08 11:43:00.687534 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:00.687507 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6702664a-2622-4663-921a-bbbecef96814-kube-api-access-gnjbp" (OuterVolumeSpecName: "kube-api-access-gnjbp") pod "6702664a-2622-4663-921a-bbbecef96814" (UID: "6702664a-2622-4663-921a-bbbecef96814"). InnerVolumeSpecName "kube-api-access-gnjbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:43:00.786607 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:00.786529 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnjbp\" (UniqueName: \"kubernetes.io/projected/6702664a-2622-4663-921a-bbbecef96814-kube-api-access-gnjbp\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:43:01.459036 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.459007 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eecf9677b571e59a07b8d448ca09c4f0c24e490196e632eda617ce5e04fdba31" Sep 08 11:43:01.459036 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.459014 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7449-140146733610160" Sep 08 11:43:01.469769 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:43:01.469735 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7449-140146733610160\" is forbidden: User \"system:node:ip-10-0-133-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-187.ec2.internal' and this object" podUID="6702664a-2622-4663-921a-bbbecef96814" pod="opendatahub/test-curl-7449-140146733610160" Sep 08 11:43:01.652478 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:43:01.652437 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7449-140146733610160\" is forbidden: User \"system:node:ip-10-0-133-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-187.ec2.internal' and this object" podUID="6702664a-2622-4663-921a-bbbecef96814" pod="opendatahub/test-curl-7449-140146733610160" Sep 08 11:43:01.653750 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.653726 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6702664a-2622-4663-921a-bbbecef96814" path="/var/lib/kubelet/pods/6702664a-2622-4663-921a-bbbecef96814/volumes" Sep 08 11:43:01.822440 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.820412 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h"] Sep 08 11:43:01.822440 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.821012 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6702664a-2622-4663-921a-bbbecef96814" containerName="test-curl-7449-140146733610160" Sep 08 11:43:01.822440 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.821032 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6702664a-2622-4663-921a-bbbecef96814" containerName="test-curl-7449-140146733610160" Sep 08 11:43:01.822440 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.821132 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6702664a-2622-4663-921a-bbbecef96814" containerName="test-curl-7449-140146733610160" Sep 08 11:43:01.824987 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.824957 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.828775 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.828359 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-a1721d-openshift-default-dockercfg-cjnhb\"" Sep 08 11:43:01.836751 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.836729 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h"] Sep 08 11:43:01.897463 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.897435 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-workload-socket\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.897602 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.897473 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-credential-socket\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.897602 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.897531 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-podinfo\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.897602 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.897569 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klsnb\" (UniqueName: \"kubernetes.io/projected/77c83f90-b1bf-4342-ac82-a6d603d881ce-kube-api-access-klsnb\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.897708 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.897628 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-data\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.897708 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.897659 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c83f90-b1bf-4342-ac82-a6d603d881ce-istiod-ca-cert\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.897708 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.897696 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-envoy\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.897804 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.897711 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-token\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.897804 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.897740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-workload-certs\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.999174 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.999136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klsnb\" (UniqueName: \"kubernetes.io/projected/77c83f90-b1bf-4342-ac82-a6d603d881ce-kube-api-access-klsnb\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.999174 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.999175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-data\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.999393 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.999195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c83f90-b1bf-4342-ac82-a6d603d881ce-istiod-ca-cert\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.999393 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.999310 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-envoy\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.999393 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.999339 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-token\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.999540 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.999405 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-workload-certs\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.999540 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.999466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-workload-socket\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.999540 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.999491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-credential-socket\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.999693 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.999550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-podinfo\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.999693 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.999622 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-data\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.999803 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.999788 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-workload-certs\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.999912 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.999886 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-workload-socket\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:01.999999 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:01.999890 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-credential-socket\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:02.000054 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:02.000033 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c83f90-b1bf-4342-ac82-a6d603d881ce-istiod-ca-cert\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:02.001534 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:02.001513 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-envoy\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:02.001865 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:02.001844 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-podinfo\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:02.008188 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:02.008161 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-token\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:02.008401 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:02.008383 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klsnb\" (UniqueName: \"kubernetes.io/projected/77c83f90-b1bf-4342-ac82-a6d603d881ce-kube-api-access-klsnb\") pod \"e2e-autores-a1721d-openshift-default-689f485974-7bh9h\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:02.138549 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:02.138518 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:02.422464 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:02.422436 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h"] Sep 08 11:43:02.424606 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:43:02.424580 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c83f90_b1bf_4342_ac82_a6d603d881ce.slice/crio-3ea1adda1dd89732fda9f46087ccc872443690d10672256829d6905837f5fab5 WatchSource:0}: Error finding container 3ea1adda1dd89732fda9f46087ccc872443690d10672256829d6905837f5fab5: Status 404 returned error can't find the container with id 3ea1adda1dd89732fda9f46087ccc872443690d10672256829d6905837f5fab5 Sep 08 11:43:02.465627 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:02.465601 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" event={"ID":"77c83f90-b1bf-4342-ac82-a6d603d881ce","Type":"ContainerStarted","Data":"3ea1adda1dd89732fda9f46087ccc872443690d10672256829d6905837f5fab5"} Sep 08 11:43:04.938969 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:04.938930 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:43:07.086861 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:07.086819 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:43:07.087167 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:07.086891 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:43:07.087167 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:07.086922 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:43:07.499755 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:07.499711 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" event={"ID":"77c83f90-b1bf-4342-ac82-a6d603d881ce","Type":"ContainerStarted","Data":"1cd388b00c33091d9795ebcc65c2fe3bd60215f9aaed9215e6c6d401dca67280"} Sep 08 11:43:07.528539 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:07.528481 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" podStartSLOduration=1.868718591 podStartE2EDuration="6.528464733s" totalImagesPullingTime="4.659746142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:43:07.527312914 +0000 UTC m=+776.386500234" watchObservedRunningTime="2026-09-08 11:43:07.528464733 +0000 UTC m=+776.387652050" Sep 08 11:43:08.138878 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:08.138844 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:08.144290 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:08.144263 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:08.505795 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:08.505714 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:08.506641 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:08.506620 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:10.617126 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.617092 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz"] Sep 08 11:43:10.620419 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.620402 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:10.623727 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.623704 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 11:43:10.624045 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.624026 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-a1721d-serving-cert\"" Sep 08 11:43:10.624149 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.624129 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 08 11:43:10.625453 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.625434 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b5t6p\"" Sep 08 11:43:10.625601 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.625581 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 08 11:43:10.634871 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.634835 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz"] Sep 08 11:43:10.783828 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.783793 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-maas-api-tls\") pod \"maas-api-e2e-autores-a1721d-7fcb489574-wwgpz\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:10.784007 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.783843 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ktps\" (UniqueName: \"kubernetes.io/projected/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-kube-api-access-2ktps\") pod \"maas-api-e2e-autores-a1721d-7fcb489574-wwgpz\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:10.784007 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.783877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-metrics-certs\") pod \"maas-api-e2e-autores-a1721d-7fcb489574-wwgpz\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:10.784111 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.784067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-openshift-service-ca\") pod \"maas-api-e2e-autores-a1721d-7fcb489574-wwgpz\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:10.884602 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.884573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-openshift-service-ca\") pod \"maas-api-e2e-autores-a1721d-7fcb489574-wwgpz\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:10.884773 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.884637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-maas-api-tls\") pod \"maas-api-e2e-autores-a1721d-7fcb489574-wwgpz\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:10.884773 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.884658 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ktps\" (UniqueName: \"kubernetes.io/projected/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-kube-api-access-2ktps\") pod \"maas-api-e2e-autores-a1721d-7fcb489574-wwgpz\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:10.884773 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.884691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-metrics-certs\") pod \"maas-api-e2e-autores-a1721d-7fcb489574-wwgpz\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:10.884773 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:43:10.884761 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-serving-cert: secret "maas-api-e2e-autores-a1721d-serving-cert" not found Sep 08 11:43:10.885005 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:43:10.884844 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-maas-api-tls podName:a9111d62-f7c8-4a52-84d8-a23d7c9362eb nodeName:}" failed. No retries permitted until 2026-09-08 11:43:11.38482126 +0000 UTC m=+780.244008561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-maas-api-tls") pod "maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" (UID: "a9111d62-f7c8-4a52-84d8-a23d7c9362eb") : secret "maas-api-e2e-autores-a1721d-serving-cert" not found Sep 08 11:43:10.885354 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.885329 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-openshift-service-ca\") pod \"maas-api-e2e-autores-a1721d-7fcb489574-wwgpz\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:10.887085 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.887067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-metrics-certs\") pod \"maas-api-e2e-autores-a1721d-7fcb489574-wwgpz\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:10.895450 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:10.895426 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ktps\" (UniqueName: \"kubernetes.io/projected/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-kube-api-access-2ktps\") pod \"maas-api-e2e-autores-a1721d-7fcb489574-wwgpz\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:11.389985 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:11.389951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-maas-api-tls\") pod \"maas-api-e2e-autores-a1721d-7fcb489574-wwgpz\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:11.392796 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:11.392770 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-maas-api-tls\") pod \"maas-api-e2e-autores-a1721d-7fcb489574-wwgpz\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:11.536096 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:11.536066 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b5t6p\"" Sep 08 11:43:11.543943 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:11.543921 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:11.821902 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:11.821872 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz"] Sep 08 11:43:11.822830 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:43:11.822807 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9111d62_f7c8_4a52_84d8_a23d7c9362eb.slice/crio-ab9a8a4f442a1f8c4c710238e73753bee1d4b960c0836698975302ea9b588cec WatchSource:0}: Error finding container ab9a8a4f442a1f8c4c710238e73753bee1d4b960c0836698975302ea9b588cec: Status 404 returned error can't find the container with id ab9a8a4f442a1f8c4c710238e73753bee1d4b960c0836698975302ea9b588cec Sep 08 11:43:12.532445 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:12.532402 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" event={"ID":"a9111d62-f7c8-4a52-84d8-a23d7c9362eb","Type":"ContainerStarted","Data":"ab9a8a4f442a1f8c4c710238e73753bee1d4b960c0836698975302ea9b588cec"} Sep 08 11:43:13.541504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:13.541243 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" event={"ID":"a9111d62-f7c8-4a52-84d8-a23d7c9362eb","Type":"ContainerStarted","Data":"574ee982d381b26f0ee7a72d9a97b3e1cb90571d29c86a46df0c7b2e14069cfa"} Sep 08 11:43:13.541504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:13.541392 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:13.572150 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:13.572101 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" podStartSLOduration=2.130991471 podStartE2EDuration="3.572083941s" totalImagesPullingTime="1.44109247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:43:13.571357561 +0000 UTC m=+782.430544899" watchObservedRunningTime="2026-09-08 11:43:13.572083941 +0000 UTC m=+782.431271262" Sep 08 11:43:19.553522 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:19.553493 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:43:19.965713 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:19.965681 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l"] Sep 08 11:43:19.973952 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:19.970181 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:19.975821 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:19.975788 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-425b2c-openshift-default-dockercfg-qnlxw\"" Sep 08 11:43:19.998468 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:19.998442 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l"] Sep 08 11:43:20.066802 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.066768 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-istio-envoy\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.066970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.066810 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-workload-socket\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.066970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.066833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-istio-data\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.066970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.066849 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6953953d-ae92-48a2-8326-841c7d40a417-istio-token\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.067109 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.066982 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6953953d-ae92-48a2-8326-841c7d40a417-istiod-ca-cert\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.067109 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.067067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-credential-socket\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.067109 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.067097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6953953d-ae92-48a2-8326-841c7d40a417-istio-podinfo\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.067232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.067123 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-workload-certs\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.067232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.067152 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nghjz\" (UniqueName: \"kubernetes.io/projected/6953953d-ae92-48a2-8326-841c7d40a417-kube-api-access-nghjz\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.168087 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.168055 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-istio-data\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.168087 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.168087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6953953d-ae92-48a2-8326-841c7d40a417-istio-token\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.168321 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.168122 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6953953d-ae92-48a2-8326-841c7d40a417-istiod-ca-cert\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.168321 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.168154 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-credential-socket\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.168321 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.168175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6953953d-ae92-48a2-8326-841c7d40a417-istio-podinfo\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.168321 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.168191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-workload-certs\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.168321 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.168209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nghjz\" (UniqueName: \"kubernetes.io/projected/6953953d-ae92-48a2-8326-841c7d40a417-kube-api-access-nghjz\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.168321 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.168274 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-istio-envoy\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.168321 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.168316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-workload-socket\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.168763 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.168607 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-istio-data\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.168821 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.168786 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-workload-certs\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.168821 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.168809 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-workload-socket\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.168894 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.168866 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-credential-socket\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.169079 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.169057 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6953953d-ae92-48a2-8326-841c7d40a417-istiod-ca-cert\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.170586 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.170564 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-istio-envoy\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.170816 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.170798 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6953953d-ae92-48a2-8326-841c7d40a417-istio-podinfo\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.178080 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.178056 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6953953d-ae92-48a2-8326-841c7d40a417-istio-token\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.178838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.178808 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nghjz\" (UniqueName: \"kubernetes.io/projected/6953953d-ae92-48a2-8326-841c7d40a417-kube-api-access-nghjz\") pod \"e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.284704 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.284622 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:20.581077 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.580994 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l"] Sep 08 11:43:20.584831 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:43:20.584803 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6953953d_ae92_48a2_8326_841c7d40a417.slice/crio-e0526e3730a5d07f8e984f0498d6ff145fc47e8b42c77d8840e89cb5f065d938 WatchSource:0}: Error finding container e0526e3730a5d07f8e984f0498d6ff145fc47e8b42c77d8840e89cb5f065d938: Status 404 returned error can't find the container with id e0526e3730a5d07f8e984f0498d6ff145fc47e8b42c77d8840e89cb5f065d938 Sep 08 11:43:20.586858 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.586822 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:43:20.586954 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.586889 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:43:20.586954 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:20.586937 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:43:21.144017 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:21.143975 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:43:21.591405 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:21.591306 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" event={"ID":"6953953d-ae92-48a2-8326-841c7d40a417","Type":"ContainerStarted","Data":"d4d08116e5a2a6177679bdc1eeab1cb8900c456ac49b8138c07bb1110ce8fcfc"} Sep 08 11:43:21.591405 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:21.591389 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" event={"ID":"6953953d-ae92-48a2-8326-841c7d40a417","Type":"ContainerStarted","Data":"e0526e3730a5d07f8e984f0498d6ff145fc47e8b42c77d8840e89cb5f065d938"} Sep 08 11:43:21.622242 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:21.622190 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" podStartSLOduration=2.6221753249999997 podStartE2EDuration="2.622175325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:43:21.622048355 +0000 UTC m=+790.481235675" watchObservedRunningTime="2026-09-08 11:43:21.622175325 +0000 UTC m=+790.481362634" Sep 08 11:43:22.285435 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:22.285390 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:22.290727 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:22.290703 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:22.598014 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:22.597931 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:22.599069 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:22.599050 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:24.904156 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:24.904122 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l"] Sep 08 11:43:24.908328 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:24.908304 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:24.910944 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:24.910922 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-425b2c-serving-cert\"" Sep 08 11:43:24.923022 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:24.923000 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l"] Sep 08 11:43:25.010232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:25.010195 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-maas-api-tls\") pod \"maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:25.010453 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:25.010241 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-metrics-certs\") pod \"maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:25.010453 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:25.010285 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vntgw\" (UniqueName: \"kubernetes.io/projected/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-kube-api-access-vntgw\") pod \"maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:25.010453 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:25.010320 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-openshift-service-ca\") pod \"maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:25.111027 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:25.110995 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-openshift-service-ca\") pod \"maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:25.111200 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:25.111115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-maas-api-tls\") pod \"maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:25.111200 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:25.111145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-metrics-certs\") pod \"maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:25.111281 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:43:25.111261 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-serving-cert: secret "maas-api-e2e-inf-b-425b2c-serving-cert" not found Sep 08 11:43:25.111340 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:43:25.111329 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-maas-api-tls podName:404a4d9c-92d4-4e9e-acbb-ce089d0f4196 nodeName:}" failed. No retries permitted until 2026-09-08 11:43:25.611312744 +0000 UTC m=+794.470500040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-maas-api-tls") pod "maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" (UID: "404a4d9c-92d4-4e9e-acbb-ce089d0f4196") : secret "maas-api-e2e-inf-b-425b2c-serving-cert" not found Sep 08 11:43:25.111426 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:25.111264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vntgw\" (UniqueName: \"kubernetes.io/projected/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-kube-api-access-vntgw\") pod \"maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:25.111802 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:25.111780 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-openshift-service-ca\") pod \"maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:25.113597 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:25.113578 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-metrics-certs\") pod \"maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:25.120666 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:25.120647 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vntgw\" (UniqueName: \"kubernetes.io/projected/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-kube-api-access-vntgw\") pod \"maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:25.615302 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:25.615264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-maas-api-tls\") pod \"maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:25.618149 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:25.618117 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-maas-api-tls\") pod \"maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:25.820799 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:25.820755 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:26.102422 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:26.102388 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l"] Sep 08 11:43:26.105852 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:43:26.105826 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404a4d9c_92d4_4e9e_acbb_ce089d0f4196.slice/crio-32a3ee13a1c5912b1a76a8c05c3f4bfe5c41a3e3bd2785ace37a1713dad5610d WatchSource:0}: Error finding container 32a3ee13a1c5912b1a76a8c05c3f4bfe5c41a3e3bd2785ace37a1713dad5610d: Status 404 returned error can't find the container with id 32a3ee13a1c5912b1a76a8c05c3f4bfe5c41a3e3bd2785ace37a1713dad5610d Sep 08 11:43:26.629339 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:26.629294 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" event={"ID":"404a4d9c-92d4-4e9e-acbb-ce089d0f4196","Type":"ContainerStarted","Data":"fc4852fb12a5baa6a8e8990d3d3b319002c8507899eb0a17ca09fd2fbfd45426"} Sep 08 11:43:26.629528 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:26.629357 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" event={"ID":"404a4d9c-92d4-4e9e-acbb-ce089d0f4196","Type":"ContainerStarted","Data":"32a3ee13a1c5912b1a76a8c05c3f4bfe5c41a3e3bd2785ace37a1713dad5610d"} Sep 08 11:43:26.629528 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:26.629500 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:26.651931 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:26.651877 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" podStartSLOduration=2.313893064 podStartE2EDuration="2.651862691s" totalImagesPullingTime="337.969627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:43:26.650548162 +0000 UTC m=+795.509735481" watchObservedRunningTime="2026-09-08 11:43:26.651862691 +0000 UTC m=+795.511050014" Sep 08 11:43:30.710545 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.710507 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r"] Sep 08 11:43:30.714179 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.714158 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.716966 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.716949 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-b2e6067b-kserve-self-signed-certs\"" Sep 08 11:43:30.747397 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.747353 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r"] Sep 08 11:43:30.768646 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.768615 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-home\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.768851 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.768814 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-dshm\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.768972 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.768873 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-tmp-dir\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.768972 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.768951 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-model-cache\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.769094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.769020 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad71e6e8-c52f-450b-aedf-52e50922443d-tls-certs\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.769094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.769046 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89rwm\" (UniqueName: \"kubernetes.io/projected/ad71e6e8-c52f-450b-aedf-52e50922443d-kube-api-access-89rwm\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.870069 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.870036 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad71e6e8-c52f-450b-aedf-52e50922443d-tls-certs\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.870069 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.870069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89rwm\" (UniqueName: \"kubernetes.io/projected/ad71e6e8-c52f-450b-aedf-52e50922443d-kube-api-access-89rwm\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.870288 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.870126 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-home\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.870288 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.870169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-dshm\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.870288 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.870200 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-tmp-dir\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.870473 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.870356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-model-cache\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.870710 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.870683 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-home\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.870840 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.870702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-tmp-dir\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.870912 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.870758 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-model-cache\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.872538 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.872517 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-dshm\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.872803 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.872777 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad71e6e8-c52f-450b-aedf-52e50922443d-tls-certs\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:30.881694 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:30.881670 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89rwm\" (UniqueName: \"kubernetes.io/projected/ad71e6e8-c52f-450b-aedf-52e50922443d-kube-api-access-89rwm\") pod \"e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:31.024905 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:31.024815 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:31.311748 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:31.311713 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r"] Sep 08 11:43:31.312725 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:43:31.312698 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad71e6e8_c52f_450b_aedf_52e50922443d.slice/crio-ecbed99efc33d026d7868b062f947f10feee7f885de458d42eab6a27358fad29 WatchSource:0}: Error finding container ecbed99efc33d026d7868b062f947f10feee7f885de458d42eab6a27358fad29: Status 404 returned error can't find the container with id ecbed99efc33d026d7868b062f947f10feee7f885de458d42eab6a27358fad29 Sep 08 11:43:31.659148 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:31.659110 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" event={"ID":"ad71e6e8-c52f-450b-aedf-52e50922443d","Type":"ContainerStarted","Data":"63773b014d969bb6f3b24eb5e586f19f6f40b46463472cd831c874b7c1ae1082"} Sep 08 11:43:31.659309 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:31.659155 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" event={"ID":"ad71e6e8-c52f-450b-aedf-52e50922443d","Type":"ContainerStarted","Data":"ecbed99efc33d026d7868b062f947f10feee7f885de458d42eab6a27358fad29"} Sep 08 11:43:31.681180 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:31.681123 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" podStartSLOduration=1.6811018930000001 podStartE2EDuration="1.681101893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:43:31.677806163 +0000 UTC m=+800.536993484" watchObservedRunningTime="2026-09-08 11:43:31.681101893 +0000 UTC m=+800.540289212" Sep 08 11:43:32.640421 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:32.640392 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:43:32.666186 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:32.666158 2564 generic.go:356] "Generic (PLEG): container finished" podID="77c83f90-b1bf-4342-ac82-a6d603d881ce" containerID="1cd388b00c33091d9795ebcc65c2fe3bd60215f9aaed9215e6c6d401dca67280" exitCode=0 Sep 08 11:43:32.666345 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:32.666237 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" event={"ID":"77c83f90-b1bf-4342-ac82-a6d603d881ce","Type":"ContainerDied","Data":"1cd388b00c33091d9795ebcc65c2fe3bd60215f9aaed9215e6c6d401dca67280"} Sep 08 11:43:32.666747 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:32.666723 2564 scope.go:117] "RemoveContainer" containerID="1cd388b00c33091d9795ebcc65c2fe3bd60215f9aaed9215e6c6d401dca67280" Sep 08 11:43:32.667942 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:32.667910 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:43:32.668071 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:32.667978 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:43:32.668071 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:32.668028 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:43:33.682444 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:33.682404 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" event={"ID":"77c83f90-b1bf-4342-ac82-a6d603d881ce","Type":"ContainerStarted","Data":"661b0752ad152ff9fb52a3f43a5fa3920072bc7e91a4d8ce9712180f5a91a15b"} Sep 08 11:43:34.139379 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:34.139334 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:34.140542 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:34.140516 2564 patch_prober.go:28] interesting pod/e2e-autores-a1721d-openshift-default-689f485974-7bh9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 08 11:43:34.140598 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:34.140569 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" podUID="77c83f90-b1bf-4342-ac82-a6d603d881ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 08 11:43:35.139020 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:35.138971 2564 patch_prober.go:28] interesting pod/e2e-autores-a1721d-openshift-default-689f485974-7bh9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 08 11:43:35.139433 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:35.139049 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" podUID="77c83f90-b1bf-4342-ac82-a6d603d881ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 08 11:43:36.139648 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:36.139609 2564 patch_prober.go:28] interesting pod/e2e-autores-a1721d-openshift-default-689f485974-7bh9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 08 11:43:36.140078 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:36.139682 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" podUID="77c83f90-b1bf-4342-ac82-a6d603d881ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 08 11:43:37.139356 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:37.139317 2564 patch_prober.go:28] interesting pod/e2e-autores-a1721d-openshift-default-689f485974-7bh9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 08 11:43:37.139552 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:37.139385 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" podUID="77c83f90-b1bf-4342-ac82-a6d603d881ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 08 11:43:38.139801 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:38.139766 2564 patch_prober.go:28] interesting pod/e2e-autores-a1721d-openshift-default-689f485974-7bh9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 08 11:43:38.140094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:38.139824 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" podUID="77c83f90-b1bf-4342-ac82-a6d603d881ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 08 11:43:38.506324 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:38.506283 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:39.139361 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:39.139317 2564 patch_prober.go:28] interesting pod/e2e-autores-a1721d-openshift-default-689f485974-7bh9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 08 11:43:39.139565 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:39.139400 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" podUID="77c83f90-b1bf-4342-ac82-a6d603d881ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 08 11:43:40.143024 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:40.142996 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:40.144027 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:40.144005 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:43:40.742627 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:40.742595 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:43:41.025363 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:41.025282 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:41.025363 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:41.025339 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:41.037854 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:41.037830 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:41.745870 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:41.745842 2564 generic.go:356] "Generic (PLEG): container finished" podID="6953953d-ae92-48a2-8326-841c7d40a417" containerID="d4d08116e5a2a6177679bdc1eeab1cb8900c456ac49b8138c07bb1110ce8fcfc" exitCode=0 Sep 08 11:43:41.746436 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:41.745929 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" event={"ID":"6953953d-ae92-48a2-8326-841c7d40a417","Type":"ContainerDied","Data":"d4d08116e5a2a6177679bdc1eeab1cb8900c456ac49b8138c07bb1110ce8fcfc"} Sep 08 11:43:41.746436 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:41.746391 2564 scope.go:117] "RemoveContainer" containerID="d4d08116e5a2a6177679bdc1eeab1cb8900c456ac49b8138c07bb1110ce8fcfc" Sep 08 11:43:41.747508 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:41.747477 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:43:41.747622 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:41.747559 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:43:41.747622 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:41.747596 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:43:41.757695 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:41.757665 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:43:42.753840 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:42.753805 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" event={"ID":"6953953d-ae92-48a2-8326-841c7d40a417","Type":"ContainerStarted","Data":"87f80cab32858a7e01d11612d5a9f41658c5d97593516d71a25b8425bce4bed7"} Sep 08 11:43:43.285300 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:43.285268 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:43.285596 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:43.285310 2564 patch_prober.go:28] interesting pod/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 08 11:43:43.285765 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:43.285741 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" podUID="6953953d-ae92-48a2-8326-841c7d40a417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 08 11:43:44.285401 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:44.285351 2564 patch_prober.go:28] interesting pod/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 08 11:43:44.285963 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:44.285932 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" podUID="6953953d-ae92-48a2-8326-841c7d40a417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 08 11:43:45.290391 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:45.287340 2564 patch_prober.go:28] interesting pod/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 08 11:43:45.290391 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:45.287427 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" podUID="6953953d-ae92-48a2-8326-841c7d40a417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 08 11:43:46.295391 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:46.287733 2564 patch_prober.go:28] interesting pod/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 08 11:43:46.295391 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:46.287814 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" podUID="6953953d-ae92-48a2-8326-841c7d40a417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 08 11:43:47.285457 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:47.285413 2564 patch_prober.go:28] interesting pod/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Sep 08 11:43:47.285659 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:47.285481 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" podUID="6953953d-ae92-48a2-8326-841c7d40a417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Sep 08 11:43:48.289485 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:48.289455 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:48.289953 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:48.289706 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:48.290481 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:48.290463 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:43:49.301333 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:49.301299 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg"] Sep 08 11:43:49.306676 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:49.306647 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:49.310341 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:49.310317 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ac040d69-serving-cert\"" Sep 08 11:43:49.318057 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:49.318035 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg"] Sep 08 11:43:49.465431 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:49.465397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22544368-010b-4287-b95d-da0a9e8e37cc-openshift-service-ca\") pod \"maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:49.465431 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:49.465441 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22544368-010b-4287-b95d-da0a9e8e37cc-metrics-certs\") pod \"maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:49.465642 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:49.465590 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22544368-010b-4287-b95d-da0a9e8e37cc-maas-api-tls\") pod \"maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:49.465717 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:49.465702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxrbv\" (UniqueName: \"kubernetes.io/projected/22544368-010b-4287-b95d-da0a9e8e37cc-kube-api-access-sxrbv\") pod \"maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:49.566665 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:49.566586 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22544368-010b-4287-b95d-da0a9e8e37cc-maas-api-tls\") pod \"maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:49.566805 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:49.566683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxrbv\" (UniqueName: \"kubernetes.io/projected/22544368-010b-4287-b95d-da0a9e8e37cc-kube-api-access-sxrbv\") pod \"maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:49.566805 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:49.566738 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22544368-010b-4287-b95d-da0a9e8e37cc-openshift-service-ca\") pod \"maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:49.566805 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:49.566765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22544368-010b-4287-b95d-da0a9e8e37cc-metrics-certs\") pod \"maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:49.566912 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:43:49.566739 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-serving-cert: secret "maas-api-e2e-migrate-ac040d69-serving-cert" not found Sep 08 11:43:49.566912 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:43:49.566872 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22544368-010b-4287-b95d-da0a9e8e37cc-maas-api-tls podName:22544368-010b-4287-b95d-da0a9e8e37cc nodeName:}" failed. No retries permitted until 2026-09-08 11:43:50.066855335 +0000 UTC m=+818.926042645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22544368-010b-4287-b95d-da0a9e8e37cc-maas-api-tls") pod "maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" (UID: "22544368-010b-4287-b95d-da0a9e8e37cc") : secret "maas-api-e2e-migrate-ac040d69-serving-cert" not found Sep 08 11:43:49.567329 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:49.567309 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22544368-010b-4287-b95d-da0a9e8e37cc-openshift-service-ca\") pod \"maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:49.569255 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:49.569232 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22544368-010b-4287-b95d-da0a9e8e37cc-metrics-certs\") pod \"maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:49.582838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:49.582809 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxrbv\" (UniqueName: \"kubernetes.io/projected/22544368-010b-4287-b95d-da0a9e8e37cc-kube-api-access-sxrbv\") pod \"maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:50.072073 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:50.072037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22544368-010b-4287-b95d-da0a9e8e37cc-maas-api-tls\") pod \"maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:50.079362 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:50.075009 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22544368-010b-4287-b95d-da0a9e8e37cc-maas-api-tls\") pod \"maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:50.220637 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:50.220595 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:50.503417 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:50.503358 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg"] Sep 08 11:43:50.504835 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:43:50.504813 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22544368_010b_4287_b95d_da0a9e8e37cc.slice/crio-a4b5bcebbb47ebc0a0a40b5d541f6cfad586f15929c94da37d06614558efe01c WatchSource:0}: Error finding container a4b5bcebbb47ebc0a0a40b5d541f6cfad586f15929c94da37d06614558efe01c: Status 404 returned error can't find the container with id a4b5bcebbb47ebc0a0a40b5d541f6cfad586f15929c94da37d06614558efe01c Sep 08 11:43:50.805353 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:50.805261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" event={"ID":"22544368-010b-4287-b95d-da0a9e8e37cc","Type":"ContainerStarted","Data":"a4b5bcebbb47ebc0a0a40b5d541f6cfad586f15929c94da37d06614558efe01c"} Sep 08 11:43:51.812938 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:51.812902 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" event={"ID":"22544368-010b-4287-b95d-da0a9e8e37cc","Type":"ContainerStarted","Data":"9f78cf561109c04aa0b6a4c05ae0615794fdcf5e867b8e18c72f06ab0c2185bd"} Sep 08 11:43:51.813419 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:51.813017 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:43:51.835531 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:51.835486 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" podStartSLOduration=2.423721397 podStartE2EDuration="2.835471579s" totalImagesPullingTime="411.750182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:43:51.833438737 +0000 UTC m=+820.692626095" watchObservedRunningTime="2026-09-08 11:43:51.835471579 +0000 UTC m=+820.694658959" Sep 08 11:43:56.442189 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:56.442155 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:43:57.824861 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:43:57.824833 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:44:11.943411 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:11.943378 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:44:20.168184 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.168148 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg"] Sep 08 11:44:20.168678 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.168418 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" podUID="22544368-010b-4287-b95d-da0a9e8e37cc" containerName="maas-api" containerID="cri-o://9f78cf561109c04aa0b6a4c05ae0615794fdcf5e867b8e18c72f06ab0c2185bd" gracePeriod=30 Sep 08 11:44:20.419669 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.419504 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:44:20.570149 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.570109 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22544368-010b-4287-b95d-da0a9e8e37cc-openshift-service-ca\") pod \"22544368-010b-4287-b95d-da0a9e8e37cc\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " Sep 08 11:44:20.570335 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.570166 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22544368-010b-4287-b95d-da0a9e8e37cc-metrics-certs\") pod \"22544368-010b-4287-b95d-da0a9e8e37cc\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " Sep 08 11:44:20.570335 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.570197 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22544368-010b-4287-b95d-da0a9e8e37cc-maas-api-tls\") pod \"22544368-010b-4287-b95d-da0a9e8e37cc\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " Sep 08 11:44:20.570488 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.570363 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxrbv\" (UniqueName: \"kubernetes.io/projected/22544368-010b-4287-b95d-da0a9e8e37cc-kube-api-access-sxrbv\") pod \"22544368-010b-4287-b95d-da0a9e8e37cc\" (UID: \"22544368-010b-4287-b95d-da0a9e8e37cc\") " Sep 08 11:44:20.570587 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.570545 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22544368-010b-4287-b95d-da0a9e8e37cc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "22544368-010b-4287-b95d-da0a9e8e37cc" (UID: "22544368-010b-4287-b95d-da0a9e8e37cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:44:20.570858 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.570830 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22544368-010b-4287-b95d-da0a9e8e37cc-openshift-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:44:20.572637 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.572603 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22544368-010b-4287-b95d-da0a9e8e37cc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "22544368-010b-4287-b95d-da0a9e8e37cc" (UID: "22544368-010b-4287-b95d-da0a9e8e37cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:44:20.572637 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.572620 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22544368-010b-4287-b95d-da0a9e8e37cc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "22544368-010b-4287-b95d-da0a9e8e37cc" (UID: "22544368-010b-4287-b95d-da0a9e8e37cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:44:20.572637 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.572603 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22544368-010b-4287-b95d-da0a9e8e37cc-kube-api-access-sxrbv" (OuterVolumeSpecName: "kube-api-access-sxrbv") pod "22544368-010b-4287-b95d-da0a9e8e37cc" (UID: "22544368-010b-4287-b95d-da0a9e8e37cc"). InnerVolumeSpecName "kube-api-access-sxrbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:44:20.672216 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.672123 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxrbv\" (UniqueName: \"kubernetes.io/projected/22544368-010b-4287-b95d-da0a9e8e37cc-kube-api-access-sxrbv\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:44:20.672216 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.672157 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22544368-010b-4287-b95d-da0a9e8e37cc-metrics-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:44:20.672216 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.672170 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22544368-010b-4287-b95d-da0a9e8e37cc-maas-api-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:44:20.984219 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.984130 2564 generic.go:356] "Generic (PLEG): container finished" podID="22544368-010b-4287-b95d-da0a9e8e37cc" containerID="9f78cf561109c04aa0b6a4c05ae0615794fdcf5e867b8e18c72f06ab0c2185bd" exitCode=0 Sep 08 11:44:20.984219 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.984181 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" Sep 08 11:44:20.984447 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.984219 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" event={"ID":"22544368-010b-4287-b95d-da0a9e8e37cc","Type":"ContainerDied","Data":"9f78cf561109c04aa0b6a4c05ae0615794fdcf5e867b8e18c72f06ab0c2185bd"} Sep 08 11:44:20.984447 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.984273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg" event={"ID":"22544368-010b-4287-b95d-da0a9e8e37cc","Type":"ContainerDied","Data":"a4b5bcebbb47ebc0a0a40b5d541f6cfad586f15929c94da37d06614558efe01c"} Sep 08 11:44:20.984447 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.984302 2564 scope.go:117] "RemoveContainer" containerID="9f78cf561109c04aa0b6a4c05ae0615794fdcf5e867b8e18c72f06ab0c2185bd" Sep 08 11:44:20.994203 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.994185 2564 scope.go:117] "RemoveContainer" containerID="9f78cf561109c04aa0b6a4c05ae0615794fdcf5e867b8e18c72f06ab0c2185bd" Sep 08 11:44:20.994482 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:44:20.994462 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f78cf561109c04aa0b6a4c05ae0615794fdcf5e867b8e18c72f06ab0c2185bd\": container with ID starting with 9f78cf561109c04aa0b6a4c05ae0615794fdcf5e867b8e18c72f06ab0c2185bd not found: ID does not exist" containerID="9f78cf561109c04aa0b6a4c05ae0615794fdcf5e867b8e18c72f06ab0c2185bd" Sep 08 11:44:20.994563 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:20.994495 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f78cf561109c04aa0b6a4c05ae0615794fdcf5e867b8e18c72f06ab0c2185bd"} err="failed to get container status \"9f78cf561109c04aa0b6a4c05ae0615794fdcf5e867b8e18c72f06ab0c2185bd\": rpc error: code = NotFound desc = could not find container \"9f78cf561109c04aa0b6a4c05ae0615794fdcf5e867b8e18c72f06ab0c2185bd\": container with ID starting with 9f78cf561109c04aa0b6a4c05ae0615794fdcf5e867b8e18c72f06ab0c2185bd not found: ID does not exist" Sep 08 11:44:21.010092 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.010068 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg"] Sep 08 11:44:21.014977 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.014956 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ac040d69-7fc5c95cf7-cbmgg"] Sep 08 11:44:21.354656 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.354559 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d"] Sep 08 11:44:21.355165 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.355144 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22544368-010b-4287-b95d-da0a9e8e37cc" containerName="maas-api" Sep 08 11:44:21.355239 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.355167 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="22544368-010b-4287-b95d-da0a9e8e37cc" containerName="maas-api" Sep 08 11:44:21.355313 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.355300 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="22544368-010b-4287-b95d-da0a9e8e37cc" containerName="maas-api" Sep 08 11:44:21.359821 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.359800 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.362681 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.362660 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b16466d7-openshift-default-dockercfg-ztjzl\"" Sep 08 11:44:21.372080 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.372056 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d"] Sep 08 11:44:21.480838 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.480802 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-data\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.481020 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.480847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-credential-socket\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.481020 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.480915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-workload-certs\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.481020 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.480971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-workload-socket\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.481020 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.480989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-podinfo\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.481020 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.481013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-token\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.481223 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.481047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62plg\" (UniqueName: \"kubernetes.io/projected/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-kube-api-access-62plg\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.481223 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.481112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-envoy\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.481223 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.481135 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istiod-ca-cert\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.582490 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.582448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-workload-socket\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.582674 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.582494 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-podinfo\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.582674 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.582527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-token\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.582674 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.582573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62plg\" (UniqueName: \"kubernetes.io/projected/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-kube-api-access-62plg\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.582855 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.582755 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-envoy\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.582855 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.582795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istiod-ca-cert\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.582855 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.582839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-data\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.583024 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.582874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-credential-socket\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.583024 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.582947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-workload-certs\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.583224 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.583203 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-data\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.583352 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.583279 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-workload-certs\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.583352 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.582874 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-workload-socket\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.583508 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.583446 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-credential-socket\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.583659 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.583602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istiod-ca-cert\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.585735 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.585699 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-envoy\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.586234 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.586211 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-podinfo\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.592930 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.592905 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62plg\" (UniqueName: \"kubernetes.io/projected/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-kube-api-access-62plg\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.593104 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.593086 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-token\") pod \"e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.653240 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.653214 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22544368-010b-4287-b95d-da0a9e8e37cc" path="/var/lib/kubelet/pods/22544368-010b-4287-b95d-da0a9e8e37cc/volumes" Sep 08 11:44:21.671429 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.671400 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:21.954712 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.954681 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d"] Sep 08 11:44:21.960576 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.960544 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:44:21.960665 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.960609 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:44:21.960665 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.960638 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:44:21.991411 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:21.991382 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" event={"ID":"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312","Type":"ContainerStarted","Data":"748dd2e0c9abfbfa88d03ad1963424a67fcb72ef57a6253b35f0cc753783f994"} Sep 08 11:44:23.000025 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:22.999988 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" event={"ID":"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312","Type":"ContainerStarted","Data":"50559cb1b3c5d2e07b455219c5d1e54b6690b0728fb53c5f534d15ce1f596e8d"} Sep 08 11:44:23.023857 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:23.023810 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" podStartSLOduration=2.023795012 podStartE2EDuration="2.023795012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:44:23.02116271 +0000 UTC m=+851.880350051" watchObservedRunningTime="2026-09-08 11:44:23.023795012 +0000 UTC m=+851.882982330" Sep 08 11:44:23.672531 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:23.672502 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:23.677564 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:23.677537 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:24.005735 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:24.005649 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:24.006767 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:24.006741 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:30.138549 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:30.138514 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:44:40.047561 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:40.047512 2564 patch_prober.go:28] interesting pod/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 11:44:40.048979 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:40.047591 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 11:44:40.110713 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:40.110685 2564 generic.go:356] "Generic (PLEG): container finished" podID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerID="50559cb1b3c5d2e07b455219c5d1e54b6690b0728fb53c5f534d15ce1f596e8d" exitCode=0 Sep 08 11:44:40.110828 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:40.110754 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" event={"ID":"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312","Type":"ContainerDied","Data":"50559cb1b3c5d2e07b455219c5d1e54b6690b0728fb53c5f534d15ce1f596e8d"} Sep 08 11:44:40.111202 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:40.111184 2564 scope.go:117] "RemoveContainer" containerID="50559cb1b3c5d2e07b455219c5d1e54b6690b0728fb53c5f534d15ce1f596e8d" Sep 08 11:44:40.112282 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:40.112253 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:44:40.112353 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:40.112338 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:44:40.112417 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:40.112393 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:44:41.118774 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:41.118744 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" event={"ID":"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312","Type":"ContainerStarted","Data":"278f9b88443d180877399f205b4d3efaad103bbb685566c5218b83e95dbc77ba"} Sep 08 11:44:41.672085 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:41.672053 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:41.672260 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:41.672223 2564 patch_prober.go:28] interesting pod/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 08 11:44:41.672311 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:41.672273 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 08 11:44:42.672164 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:42.672132 2564 patch_prober.go:28] interesting pod/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 08 11:44:42.672566 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:42.672195 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 08 11:44:43.671750 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:43.671712 2564 patch_prober.go:28] interesting pod/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 08 11:44:43.671949 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:43.671772 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 08 11:44:44.672287 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:44.672251 2564 patch_prober.go:28] interesting pod/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 08 11:44:44.672698 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:44.672305 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 08 11:44:45.672248 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:45.672215 2564 patch_prober.go:28] interesting pod/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 08 11:44:45.672430 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:45.672273 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 08 11:44:46.672763 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:46.672726 2564 patch_prober.go:28] interesting pod/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 08 11:44:46.673138 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:46.672778 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 08 11:44:47.675615 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:47.675582 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:47.675986 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:47.675970 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:47.676630 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:47.676613 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:44:51.832040 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:51.832006 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r"] Sep 08 11:44:52.044550 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:52.044516 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:44:54.010420 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.010381 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz"] Sep 08 11:44:54.010866 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.010643 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" podUID="a9111d62-f7c8-4a52-84d8-a23d7c9362eb" containerName="maas-api" containerID="cri-o://574ee982d381b26f0ee7a72d9a97b3e1cb90571d29c86a46df0c7b2e14069cfa" gracePeriod=30 Sep 08 11:44:54.215937 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.215855 2564 generic.go:356] "Generic (PLEG): container finished" podID="a9111d62-f7c8-4a52-84d8-a23d7c9362eb" containerID="574ee982d381b26f0ee7a72d9a97b3e1cb90571d29c86a46df0c7b2e14069cfa" exitCode=0 Sep 08 11:44:54.216072 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.215963 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" event={"ID":"a9111d62-f7c8-4a52-84d8-a23d7c9362eb","Type":"ContainerDied","Data":"574ee982d381b26f0ee7a72d9a97b3e1cb90571d29c86a46df0c7b2e14069cfa"} Sep 08 11:44:54.280303 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.280230 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:44:54.433854 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.433817 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-metrics-certs\") pod \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " Sep 08 11:44:54.434038 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.433894 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ktps\" (UniqueName: \"kubernetes.io/projected/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-kube-api-access-2ktps\") pod \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " Sep 08 11:44:54.434038 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.433947 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-openshift-service-ca\") pod \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " Sep 08 11:44:54.434163 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.434039 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-maas-api-tls\") pod \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\" (UID: \"a9111d62-f7c8-4a52-84d8-a23d7c9362eb\") " Sep 08 11:44:54.434348 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.434314 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a9111d62-f7c8-4a52-84d8-a23d7c9362eb" (UID: "a9111d62-f7c8-4a52-84d8-a23d7c9362eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:44:54.434496 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.434444 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-openshift-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:44:54.436037 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.436005 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a9111d62-f7c8-4a52-84d8-a23d7c9362eb" (UID: "a9111d62-f7c8-4a52-84d8-a23d7c9362eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:44:54.436126 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.436054 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a9111d62-f7c8-4a52-84d8-a23d7c9362eb" (UID: "a9111d62-f7c8-4a52-84d8-a23d7c9362eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:44:54.436126 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.436085 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-kube-api-access-2ktps" (OuterVolumeSpecName: "kube-api-access-2ktps") pod "a9111d62-f7c8-4a52-84d8-a23d7c9362eb" (UID: "a9111d62-f7c8-4a52-84d8-a23d7c9362eb"). InnerVolumeSpecName "kube-api-access-2ktps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:44:54.535650 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.535593 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-maas-api-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:44:54.535650 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.535614 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-metrics-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:44:54.535650 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:54.535624 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ktps\" (UniqueName: \"kubernetes.io/projected/a9111d62-f7c8-4a52-84d8-a23d7c9362eb-kube-api-access-2ktps\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:44:55.223755 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:55.223724 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" Sep 08 11:44:55.224172 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:55.223754 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz" event={"ID":"a9111d62-f7c8-4a52-84d8-a23d7c9362eb","Type":"ContainerDied","Data":"ab9a8a4f442a1f8c4c710238e73753bee1d4b960c0836698975302ea9b588cec"} Sep 08 11:44:55.224172 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:55.223821 2564 scope.go:117] "RemoveContainer" containerID="574ee982d381b26f0ee7a72d9a97b3e1cb90571d29c86a46df0c7b2e14069cfa" Sep 08 11:44:55.249673 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:55.249645 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz"] Sep 08 11:44:55.255480 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:55.255454 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a1721d-7fcb489574-wwgpz"] Sep 08 11:44:55.653209 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:55.653175 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9111d62-f7c8-4a52-84d8-a23d7c9362eb" path="/var/lib/kubelet/pods/a9111d62-f7c8-4a52-84d8-a23d7c9362eb/volumes" Sep 08 11:44:57.859263 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:57.859232 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d"] Sep 08 11:44:57.859731 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:44:57.859516 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerName="istio-proxy" containerID="cri-o://278f9b88443d180877399f205b4d3efaad103bbb685566c5218b83e95dbc77ba" gracePeriod=30 Sep 08 11:45:00.149427 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:00.149388 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-95669b-29814465-xbbz9"] Sep 08 11:45:00.149821 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:00.149802 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9111d62-f7c8-4a52-84d8-a23d7c9362eb" containerName="maas-api" Sep 08 11:45:00.149821 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:00.149813 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9111d62-f7c8-4a52-84d8-a23d7c9362eb" containerName="maas-api" Sep 08 11:45:00.149894 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:00.149887 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9111d62-f7c8-4a52-84d8-a23d7c9362eb" containerName="maas-api" Sep 08 11:45:00.152727 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:00.152696 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-95669b-29814465-xbbz9" Sep 08 11:45:00.156010 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:00.155986 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\"" Sep 08 11:45:00.165115 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:00.165071 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-95669b-29814465-xbbz9"] Sep 08 11:45:00.189606 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:00.189587 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-95669b-29814465-xbbz9" Sep 08 11:45:00.469182 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:00.469150 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-95669b-29814465-xbbz9"] Sep 08 11:45:00.469182 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:45:00.469174 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ef437b_5d04_48bf_b696_1bd6218d5571.slice/crio-16308daade27c8007c7c7eb590c50b3f98dd4272385f51b3a9b259b845bc8584 WatchSource:0}: Error finding container 16308daade27c8007c7c7eb590c50b3f98dd4272385f51b3a9b259b845bc8584: Status 404 returned error can't find the container with id 16308daade27c8007c7c7eb590c50b3f98dd4272385f51b3a9b259b845bc8584 Sep 08 11:45:01.267764 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:01.267723 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-95669b-29814465-xbbz9" event={"ID":"e0ef437b-5d04-48bf-b696-1bd6218d5571","Type":"ContainerStarted","Data":"16308daade27c8007c7c7eb590c50b3f98dd4272385f51b3a9b259b845bc8584"} Sep 08 11:45:02.678787 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:02.678714 2564 patch_prober.go:28] interesting pod/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 08 11:45:02.679222 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:02.678797 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 11:45:03.124703 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.124671 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h"] Sep 08 11:45:03.124957 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.124904 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" podUID="77c83f90-b1bf-4342-ac82-a6d603d881ce" containerName="istio-proxy" containerID="cri-o://661b0752ad152ff9fb52a3f43a5fa3920072bc7e91a4d8ce9712180f5a91a15b" gracePeriod=30 Sep 08 11:45:03.164900 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.164879 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:45:03.224192 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.224161 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-data\") pod \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " Sep 08 11:45:03.224192 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.224192 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-envoy\") pod \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " Sep 08 11:45:03.224487 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.224227 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-podinfo\") pod \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " Sep 08 11:45:03.224487 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.224266 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-workload-socket\") pod \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " Sep 08 11:45:03.224487 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.224321 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-credential-socket\") pod \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " Sep 08 11:45:03.224487 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.224407 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istiod-ca-cert\") pod \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " Sep 08 11:45:03.224487 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.224439 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-workload-certs\") pod \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " Sep 08 11:45:03.224487 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.224482 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-token\") pod \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " Sep 08 11:45:03.224829 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.224517 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62plg\" (UniqueName: \"kubernetes.io/projected/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-kube-api-access-62plg\") pod \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\" (UID: \"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312\") " Sep 08 11:45:03.224829 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.224502 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" (UID: "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:45:03.224829 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.224733 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" (UID: "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:45:03.224994 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.224901 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" (UID: "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:45:03.225049 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.225039 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istiod-ca-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:03.225106 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.225055 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-data\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:03.225106 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.225069 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-credential-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:03.225217 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.225143 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" (UID: "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:45:03.225217 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.225168 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" (UID: "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:45:03.226843 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.226818 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" (UID: "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:45:03.226965 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.226943 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" (UID: "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 11:45:03.227144 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.227124 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-kube-api-access-62plg" (OuterVolumeSpecName: "kube-api-access-62plg") pod "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" (UID: "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312"). InnerVolumeSpecName "kube-api-access-62plg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:45:03.227406 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.227356 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" (UID: "f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:45:03.283066 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.283000 2564 generic.go:356] "Generic (PLEG): container finished" podID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerID="278f9b88443d180877399f205b4d3efaad103bbb685566c5218b83e95dbc77ba" exitCode=0 Sep 08 11:45:03.283164 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.283066 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" Sep 08 11:45:03.283164 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.283075 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" event={"ID":"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312","Type":"ContainerDied","Data":"278f9b88443d180877399f205b4d3efaad103bbb685566c5218b83e95dbc77ba"} Sep 08 11:45:03.283164 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.283116 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d" event={"ID":"f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312","Type":"ContainerDied","Data":"748dd2e0c9abfbfa88d03ad1963424a67fcb72ef57a6253b35f0cc753783f994"} Sep 08 11:45:03.283164 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.283137 2564 scope.go:117] "RemoveContainer" containerID="278f9b88443d180877399f205b4d3efaad103bbb685566c5218b83e95dbc77ba" Sep 08 11:45:03.293403 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.293354 2564 scope.go:117] "RemoveContainer" containerID="50559cb1b3c5d2e07b455219c5d1e54b6690b0728fb53c5f534d15ce1f596e8d" Sep 08 11:45:03.301944 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.301928 2564 scope.go:117] "RemoveContainer" containerID="278f9b88443d180877399f205b4d3efaad103bbb685566c5218b83e95dbc77ba" Sep 08 11:45:03.302210 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:45:03.302192 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"278f9b88443d180877399f205b4d3efaad103bbb685566c5218b83e95dbc77ba\": container with ID starting with 278f9b88443d180877399f205b4d3efaad103bbb685566c5218b83e95dbc77ba not found: ID does not exist" containerID="278f9b88443d180877399f205b4d3efaad103bbb685566c5218b83e95dbc77ba" Sep 08 11:45:03.302291 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.302223 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278f9b88443d180877399f205b4d3efaad103bbb685566c5218b83e95dbc77ba"} err="failed to get container status \"278f9b88443d180877399f205b4d3efaad103bbb685566c5218b83e95dbc77ba\": rpc error: code = NotFound desc = could not find container \"278f9b88443d180877399f205b4d3efaad103bbb685566c5218b83e95dbc77ba\": container with ID starting with 278f9b88443d180877399f205b4d3efaad103bbb685566c5218b83e95dbc77ba not found: ID does not exist" Sep 08 11:45:03.302291 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.302249 2564 scope.go:117] "RemoveContainer" containerID="50559cb1b3c5d2e07b455219c5d1e54b6690b0728fb53c5f534d15ce1f596e8d" Sep 08 11:45:03.302544 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:45:03.302527 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50559cb1b3c5d2e07b455219c5d1e54b6690b0728fb53c5f534d15ce1f596e8d\": container with ID starting with 50559cb1b3c5d2e07b455219c5d1e54b6690b0728fb53c5f534d15ce1f596e8d not found: ID does not exist" containerID="50559cb1b3c5d2e07b455219c5d1e54b6690b0728fb53c5f534d15ce1f596e8d" Sep 08 11:45:03.302594 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.302549 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50559cb1b3c5d2e07b455219c5d1e54b6690b0728fb53c5f534d15ce1f596e8d"} err="failed to get container status \"50559cb1b3c5d2e07b455219c5d1e54b6690b0728fb53c5f534d15ce1f596e8d\": rpc error: code = NotFound desc = could not find container \"50559cb1b3c5d2e07b455219c5d1e54b6690b0728fb53c5f534d15ce1f596e8d\": container with ID starting with 50559cb1b3c5d2e07b455219c5d1e54b6690b0728fb53c5f534d15ce1f596e8d not found: ID does not exist" Sep 08 11:45:03.311540 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.311519 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d"] Sep 08 11:45:03.315020 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.314992 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b16466d7-openshift-default-dd465fcd4-v2t8d"] Sep 08 11:45:03.325967 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.325945 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-workload-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:03.325967 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.325965 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:03.326082 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.325973 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62plg\" (UniqueName: \"kubernetes.io/projected/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-kube-api-access-62plg\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:03.326082 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.325982 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-envoy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:03.326082 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.325992 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-istio-podinfo\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:03.326082 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.326000 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312-workload-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:03.653606 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.653571 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" path="/var/lib/kubelet/pods/f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312/volumes" Sep 08 11:45:03.846041 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.846011 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q"] Sep 08 11:45:03.846460 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.846446 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerName="istio-proxy" Sep 08 11:45:03.846510 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.846463 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerName="istio-proxy" Sep 08 11:45:03.846510 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.846477 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerName="istio-proxy" Sep 08 11:45:03.846510 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.846482 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerName="istio-proxy" Sep 08 11:45:03.846607 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.846555 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerName="istio-proxy" Sep 08 11:45:03.846607 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.846564 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b3a52f-03d7-4bd1-aa8f-94b1d39ab312" containerName="istio-proxy" Sep 08 11:45:03.850027 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.850012 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:03.852605 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.852583 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-20cf96-openshift-default-dockercfg-rkl92\"" Sep 08 11:45:03.861223 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.861199 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q"] Sep 08 11:45:03.932941 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.932856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-workload-certs\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:03.932941 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.932893 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istiod-ca-cert\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:03.932941 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.932917 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnmtp\" (UniqueName: \"kubernetes.io/projected/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-kube-api-access-pnmtp\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:03.933151 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.932981 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-token\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:03.933151 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.933049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-workload-socket\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:03.933151 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.933090 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-credential-socket\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:03.933151 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.933110 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-data\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:03.933277 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.933166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-podinfo\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:03.933277 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:03.933200 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-envoy\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.034039 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.033998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-workload-socket\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.034228 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.034051 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-credential-socket\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.034228 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.034082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-data\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.034228 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.034139 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-podinfo\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.034228 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.034174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-envoy\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.034469 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.034271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-workload-certs\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.034469 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.034305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istiod-ca-cert\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.034469 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.034341 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnmtp\" (UniqueName: \"kubernetes.io/projected/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-kube-api-access-pnmtp\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.034469 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.034409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-token\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.034675 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.034489 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-workload-socket\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.034675 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.034528 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-credential-socket\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.034812 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.034788 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-workload-certs\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.035057 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.035033 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-data\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.035162 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.035145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istiod-ca-cert\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.036901 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.036875 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-envoy\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.037792 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.037758 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-podinfo\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.045902 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.045459 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-token\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.051285 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.046606 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnmtp\" (UniqueName: \"kubernetes.io/projected/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-kube-api-access-pnmtp\") pod \"e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.162560 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.162524 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:04.290010 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.289975 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0ef437b-5d04-48bf-b696-1bd6218d5571" containerID="742e0abdb3b26a6d90effc8dd60c1454c8de80ba5061c570b7dc8e20b7cf6825" exitCode=0 Sep 08 11:45:04.290178 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.290054 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-95669b-29814465-xbbz9" event={"ID":"e0ef437b-5d04-48bf-b696-1bd6218d5571","Type":"ContainerDied","Data":"742e0abdb3b26a6d90effc8dd60c1454c8de80ba5061c570b7dc8e20b7cf6825"} Sep 08 11:45:04.448836 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.448812 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q"] Sep 08 11:45:04.449039 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:45:04.449019 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fcb63a2_0dd6_4a6e_b98b_f5200f947ca4.slice/crio-a7fa3dcefce3c25d6b6cff4ab22ff3694ad4f8a1235262bd99a5cd391706a7a4 WatchSource:0}: Error finding container a7fa3dcefce3c25d6b6cff4ab22ff3694ad4f8a1235262bd99a5cd391706a7a4: Status 404 returned error can't find the container with id a7fa3dcefce3c25d6b6cff4ab22ff3694ad4f8a1235262bd99a5cd391706a7a4 Sep 08 11:45:04.451282 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.451253 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:45:04.451345 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.451312 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:45:04.451395 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:04.451342 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:45:05.301870 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:05.301827 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" event={"ID":"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4","Type":"ContainerStarted","Data":"45190baa80187399eff43cf15069b9efa4c98449ab5ec5aeccdeae87e66427fd"} Sep 08 11:45:05.302296 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:05.301887 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" event={"ID":"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4","Type":"ContainerStarted","Data":"a7fa3dcefce3c25d6b6cff4ab22ff3694ad4f8a1235262bd99a5cd391706a7a4"} Sep 08 11:45:05.325112 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:05.325062 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" podStartSLOduration=2.3250465719999998 podStartE2EDuration="2.325046572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:45:05.323837421 +0000 UTC m=+894.183024740" watchObservedRunningTime="2026-09-08 11:45:05.325046572 +0000 UTC m=+894.184233889" Sep 08 11:45:05.447261 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:05.447235 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-95669b-29814465-xbbz9" Sep 08 11:45:06.163255 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:06.163219 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:06.168432 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:06.168409 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:06.309586 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:06.309555 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-95669b-29814465-xbbz9" Sep 08 11:45:06.309998 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:06.309592 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-95669b-29814465-xbbz9" event={"ID":"e0ef437b-5d04-48bf-b696-1bd6218d5571","Type":"ContainerDied","Data":"16308daade27c8007c7c7eb590c50b3f98dd4272385f51b3a9b259b845bc8584"} Sep 08 11:45:06.309998 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:06.309635 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16308daade27c8007c7c7eb590c50b3f98dd4272385f51b3a9b259b845bc8584" Sep 08 11:45:06.309998 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:06.309959 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:06.310913 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:06.310891 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:06.851570 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:06.851529 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" podUID="ad71e6e8-c52f-450b-aedf-52e50922443d" containerName="main" containerID="cri-o://63773b014d969bb6f3b24eb5e586f19f6f40b46463472cd831c874b7c1ae1082" gracePeriod=45 Sep 08 11:45:07.099792 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.099765 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:45:07.162836 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.162808 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89rwm\" (UniqueName: \"kubernetes.io/projected/ad71e6e8-c52f-450b-aedf-52e50922443d-kube-api-access-89rwm\") pod \"ad71e6e8-c52f-450b-aedf-52e50922443d\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " Sep 08 11:45:07.162975 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.162854 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-model-cache\") pod \"ad71e6e8-c52f-450b-aedf-52e50922443d\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " Sep 08 11:45:07.162975 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.162895 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-home\") pod \"ad71e6e8-c52f-450b-aedf-52e50922443d\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " Sep 08 11:45:07.163062 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.162985 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad71e6e8-c52f-450b-aedf-52e50922443d-tls-certs\") pod \"ad71e6e8-c52f-450b-aedf-52e50922443d\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " Sep 08 11:45:07.163062 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.163053 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-dshm\") pod \"ad71e6e8-c52f-450b-aedf-52e50922443d\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " Sep 08 11:45:07.163155 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.163116 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad71e6e8-c52f-450b-aedf-52e50922443d" (UID: "ad71e6e8-c52f-450b-aedf-52e50922443d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:45:07.163155 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.163130 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-home" (OuterVolumeSpecName: "home") pod "ad71e6e8-c52f-450b-aedf-52e50922443d" (UID: "ad71e6e8-c52f-450b-aedf-52e50922443d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:45:07.163155 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.163140 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-tmp-dir\") pod \"ad71e6e8-c52f-450b-aedf-52e50922443d\" (UID: \"ad71e6e8-c52f-450b-aedf-52e50922443d\") " Sep 08 11:45:07.163396 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.163347 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad71e6e8-c52f-450b-aedf-52e50922443d" (UID: "ad71e6e8-c52f-450b-aedf-52e50922443d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:45:07.163529 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.163497 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:07.163529 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.163513 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:07.163529 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.163526 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:07.165077 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.165054 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad71e6e8-c52f-450b-aedf-52e50922443d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad71e6e8-c52f-450b-aedf-52e50922443d" (UID: "ad71e6e8-c52f-450b-aedf-52e50922443d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:45:07.165203 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.165166 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-dshm" (OuterVolumeSpecName: "dshm") pod "ad71e6e8-c52f-450b-aedf-52e50922443d" (UID: "ad71e6e8-c52f-450b-aedf-52e50922443d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:45:07.165203 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.165175 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad71e6e8-c52f-450b-aedf-52e50922443d-kube-api-access-89rwm" (OuterVolumeSpecName: "kube-api-access-89rwm") pod "ad71e6e8-c52f-450b-aedf-52e50922443d" (UID: "ad71e6e8-c52f-450b-aedf-52e50922443d"). InnerVolumeSpecName "kube-api-access-89rwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:45:07.264794 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.264761 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89rwm\" (UniqueName: \"kubernetes.io/projected/ad71e6e8-c52f-450b-aedf-52e50922443d-kube-api-access-89rwm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:07.264794 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.264791 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad71e6e8-c52f-450b-aedf-52e50922443d-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:07.264794 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.264800 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad71e6e8-c52f-450b-aedf-52e50922443d-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:07.316795 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.316761 2564 generic.go:356] "Generic (PLEG): container finished" podID="ad71e6e8-c52f-450b-aedf-52e50922443d" containerID="63773b014d969bb6f3b24eb5e586f19f6f40b46463472cd831c874b7c1ae1082" exitCode=0 Sep 08 11:45:07.317243 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.316807 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" Sep 08 11:45:07.317243 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.316846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" event={"ID":"ad71e6e8-c52f-450b-aedf-52e50922443d","Type":"ContainerDied","Data":"63773b014d969bb6f3b24eb5e586f19f6f40b46463472cd831c874b7c1ae1082"} Sep 08 11:45:07.317243 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.316899 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r" event={"ID":"ad71e6e8-c52f-450b-aedf-52e50922443d","Type":"ContainerDied","Data":"ecbed99efc33d026d7868b062f947f10feee7f885de458d42eab6a27358fad29"} Sep 08 11:45:07.317243 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.316933 2564 scope.go:117] "RemoveContainer" containerID="63773b014d969bb6f3b24eb5e586f19f6f40b46463472cd831c874b7c1ae1082" Sep 08 11:45:07.326556 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.326539 2564 scope.go:117] "RemoveContainer" containerID="63773b014d969bb6f3b24eb5e586f19f6f40b46463472cd831c874b7c1ae1082" Sep 08 11:45:07.326827 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:45:07.326805 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63773b014d969bb6f3b24eb5e586f19f6f40b46463472cd831c874b7c1ae1082\": container with ID starting with 63773b014d969bb6f3b24eb5e586f19f6f40b46463472cd831c874b7c1ae1082 not found: ID does not exist" containerID="63773b014d969bb6f3b24eb5e586f19f6f40b46463472cd831c874b7c1ae1082" Sep 08 11:45:07.326879 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.326836 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63773b014d969bb6f3b24eb5e586f19f6f40b46463472cd831c874b7c1ae1082"} err="failed to get container status \"63773b014d969bb6f3b24eb5e586f19f6f40b46463472cd831c874b7c1ae1082\": rpc error: code = NotFound desc = could not find container \"63773b014d969bb6f3b24eb5e586f19f6f40b46463472cd831c874b7c1ae1082\": container with ID starting with 63773b014d969bb6f3b24eb5e586f19f6f40b46463472cd831c874b7c1ae1082 not found: ID does not exist" Sep 08 11:45:07.340734 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.340713 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r"] Sep 08 11:45:07.345875 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.345854 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-b2e6067b-kserve-7fbb8759fc-rln6r"] Sep 08 11:45:07.654188 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:07.654153 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad71e6e8-c52f-450b-aedf-52e50922443d" path="/var/lib/kubelet/pods/ad71e6e8-c52f-450b-aedf-52e50922443d/volumes" Sep 08 11:45:08.327276 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.327244 2564 generic.go:356] "Generic (PLEG): container finished" podID="77c83f90-b1bf-4342-ac82-a6d603d881ce" containerID="661b0752ad152ff9fb52a3f43a5fa3920072bc7e91a4d8ce9712180f5a91a15b" exitCode=0 Sep 08 11:45:08.327740 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.327316 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" event={"ID":"77c83f90-b1bf-4342-ac82-a6d603d881ce","Type":"ContainerDied","Data":"661b0752ad152ff9fb52a3f43a5fa3920072bc7e91a4d8ce9712180f5a91a15b"} Sep 08 11:45:08.327740 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.327410 2564 scope.go:117] "RemoveContainer" containerID="1cd388b00c33091d9795ebcc65c2fe3bd60215f9aaed9215e6c6d401dca67280" Sep 08 11:45:08.379496 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.379472 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:45:08.403859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.403780 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v"] Sep 08 11:45:08.404391 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.404354 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77c83f90-b1bf-4342-ac82-a6d603d881ce" containerName="istio-proxy" Sep 08 11:45:08.404391 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.404386 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="77c83f90-b1bf-4342-ac82-a6d603d881ce" containerName="istio-proxy" Sep 08 11:45:08.404514 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.404412 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ef437b-5d04-48bf-b696-1bd6218d5571" containerName="cleanup" Sep 08 11:45:08.404514 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.404421 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ef437b-5d04-48bf-b696-1bd6218d5571" containerName="cleanup" Sep 08 11:45:08.404514 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.404433 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad71e6e8-c52f-450b-aedf-52e50922443d" containerName="main" Sep 08 11:45:08.404514 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.404441 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad71e6e8-c52f-450b-aedf-52e50922443d" containerName="main" Sep 08 11:45:08.404640 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.404533 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0ef437b-5d04-48bf-b696-1bd6218d5571" containerName="cleanup" Sep 08 11:45:08.404640 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.404544 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="77c83f90-b1bf-4342-ac82-a6d603d881ce" containerName="istio-proxy" Sep 08 11:45:08.404640 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.404557 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad71e6e8-c52f-450b-aedf-52e50922443d" containerName="main" Sep 08 11:45:08.407911 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.407894 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:08.410967 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.410947 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-20cf96-serving-cert\"" Sep 08 11:45:08.421980 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.421952 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v"] Sep 08 11:45:08.475954 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.475924 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c83f90-b1bf-4342-ac82-a6d603d881ce-istiod-ca-cert\") pod \"77c83f90-b1bf-4342-ac82-a6d603d881ce\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " Sep 08 11:45:08.476142 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.475971 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-podinfo\") pod \"77c83f90-b1bf-4342-ac82-a6d603d881ce\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " Sep 08 11:45:08.476142 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.475991 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-envoy\") pod \"77c83f90-b1bf-4342-ac82-a6d603d881ce\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " Sep 08 11:45:08.476142 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476037 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-data\") pod \"77c83f90-b1bf-4342-ac82-a6d603d881ce\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " Sep 08 11:45:08.476142 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476054 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-workload-socket\") pod \"77c83f90-b1bf-4342-ac82-a6d603d881ce\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " Sep 08 11:45:08.476142 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476074 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klsnb\" (UniqueName: \"kubernetes.io/projected/77c83f90-b1bf-4342-ac82-a6d603d881ce-kube-api-access-klsnb\") pod \"77c83f90-b1bf-4342-ac82-a6d603d881ce\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " Sep 08 11:45:08.476142 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476097 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-credential-socket\") pod \"77c83f90-b1bf-4342-ac82-a6d603d881ce\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " Sep 08 11:45:08.476142 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476125 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-token\") pod \"77c83f90-b1bf-4342-ac82-a6d603d881ce\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " Sep 08 11:45:08.476558 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476172 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-workload-certs\") pod \"77c83f90-b1bf-4342-ac82-a6d603d881ce\" (UID: \"77c83f90-b1bf-4342-ac82-a6d603d881ce\") " Sep 08 11:45:08.476558 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476355 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a616290-eeb8-4957-adc5-233dbf6b6973-maas-api-tls\") pod \"maas-api-e2e-explicit-20cf96-9497dd555-wbc6v\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:08.476558 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476435 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49flk\" (UniqueName: \"kubernetes.io/projected/7a616290-eeb8-4957-adc5-233dbf6b6973-kube-api-access-49flk\") pod \"maas-api-e2e-explicit-20cf96-9497dd555-wbc6v\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:08.476558 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476437 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "77c83f90-b1bf-4342-ac82-a6d603d881ce" (UID: "77c83f90-b1bf-4342-ac82-a6d603d881ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:45:08.476558 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476461 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c83f90-b1bf-4342-ac82-a6d603d881ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "77c83f90-b1bf-4342-ac82-a6d603d881ce" (UID: "77c83f90-b1bf-4342-ac82-a6d603d881ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:45:08.476558 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476474 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "77c83f90-b1bf-4342-ac82-a6d603d881ce" (UID: "77c83f90-b1bf-4342-ac82-a6d603d881ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:45:08.476558 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476504 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a616290-eeb8-4957-adc5-233dbf6b6973-openshift-service-ca\") pod \"maas-api-e2e-explicit-20cf96-9497dd555-wbc6v\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:08.476558 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a616290-eeb8-4957-adc5-233dbf6b6973-metrics-certs\") pod \"maas-api-e2e-explicit-20cf96-9497dd555-wbc6v\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:08.477046 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476642 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "77c83f90-b1bf-4342-ac82-a6d603d881ce" (UID: "77c83f90-b1bf-4342-ac82-a6d603d881ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:45:08.477046 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476739 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77c83f90-b1bf-4342-ac82-a6d603d881ce-istiod-ca-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:08.477046 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476757 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-data\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:08.477046 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476771 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-credential-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:08.477046 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476785 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-workload-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:08.477046 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.476844 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "77c83f90-b1bf-4342-ac82-a6d603d881ce" (UID: "77c83f90-b1bf-4342-ac82-a6d603d881ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:45:08.478566 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.478542 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "77c83f90-b1bf-4342-ac82-a6d603d881ce" (UID: "77c83f90-b1bf-4342-ac82-a6d603d881ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 11:45:08.478680 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.478576 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c83f90-b1bf-4342-ac82-a6d603d881ce-kube-api-access-klsnb" (OuterVolumeSpecName: "kube-api-access-klsnb") pod "77c83f90-b1bf-4342-ac82-a6d603d881ce" (UID: "77c83f90-b1bf-4342-ac82-a6d603d881ce"). InnerVolumeSpecName "kube-api-access-klsnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:45:08.478680 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.478605 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "77c83f90-b1bf-4342-ac82-a6d603d881ce" (UID: "77c83f90-b1bf-4342-ac82-a6d603d881ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:45:08.478762 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.478728 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "77c83f90-b1bf-4342-ac82-a6d603d881ce" (UID: "77c83f90-b1bf-4342-ac82-a6d603d881ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:45:08.577584 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.577542 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a616290-eeb8-4957-adc5-233dbf6b6973-maas-api-tls\") pod \"maas-api-e2e-explicit-20cf96-9497dd555-wbc6v\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:08.577764 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.577610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49flk\" (UniqueName: \"kubernetes.io/projected/7a616290-eeb8-4957-adc5-233dbf6b6973-kube-api-access-49flk\") pod \"maas-api-e2e-explicit-20cf96-9497dd555-wbc6v\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:08.577764 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.577641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a616290-eeb8-4957-adc5-233dbf6b6973-openshift-service-ca\") pod \"maas-api-e2e-explicit-20cf96-9497dd555-wbc6v\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:08.577764 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.577661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a616290-eeb8-4957-adc5-233dbf6b6973-metrics-certs\") pod \"maas-api-e2e-explicit-20cf96-9497dd555-wbc6v\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:08.577764 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:45:08.577710 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-serving-cert: secret "maas-api-e2e-explicit-20cf96-serving-cert" not found Sep 08 11:45:08.577990 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:45:08.577787 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a616290-eeb8-4957-adc5-233dbf6b6973-maas-api-tls podName:7a616290-eeb8-4957-adc5-233dbf6b6973 nodeName:}" failed. No retries permitted until 2026-09-08 11:45:09.077764124 +0000 UTC m=+897.936951434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a616290-eeb8-4957-adc5-233dbf6b6973-maas-api-tls") pod "maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" (UID: "7a616290-eeb8-4957-adc5-233dbf6b6973") : secret "maas-api-e2e-explicit-20cf96-serving-cert" not found Sep 08 11:45:08.577990 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.577812 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-podinfo\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:08.577990 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.577829 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-envoy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:08.577990 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.577844 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77c83f90-b1bf-4342-ac82-a6d603d881ce-workload-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:08.577990 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.577858 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klsnb\" (UniqueName: \"kubernetes.io/projected/77c83f90-b1bf-4342-ac82-a6d603d881ce-kube-api-access-klsnb\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:08.577990 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.577871 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77c83f90-b1bf-4342-ac82-a6d603d881ce-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:45:08.578475 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.578451 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a616290-eeb8-4957-adc5-233dbf6b6973-openshift-service-ca\") pod \"maas-api-e2e-explicit-20cf96-9497dd555-wbc6v\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:08.580185 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.580168 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a616290-eeb8-4957-adc5-233dbf6b6973-metrics-certs\") pod \"maas-api-e2e-explicit-20cf96-9497dd555-wbc6v\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:08.587278 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:08.587259 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49flk\" (UniqueName: \"kubernetes.io/projected/7a616290-eeb8-4957-adc5-233dbf6b6973-kube-api-access-49flk\") pod \"maas-api-e2e-explicit-20cf96-9497dd555-wbc6v\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:09.082482 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:09.082441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a616290-eeb8-4957-adc5-233dbf6b6973-maas-api-tls\") pod \"maas-api-e2e-explicit-20cf96-9497dd555-wbc6v\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:09.084786 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:09.084766 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a616290-eeb8-4957-adc5-233dbf6b6973-maas-api-tls\") pod \"maas-api-e2e-explicit-20cf96-9497dd555-wbc6v\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:09.322590 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:09.322542 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:09.335984 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:09.335895 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" Sep 08 11:45:09.335984 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:09.335947 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h" event={"ID":"77c83f90-b1bf-4342-ac82-a6d603d881ce","Type":"ContainerDied","Data":"3ea1adda1dd89732fda9f46087ccc872443690d10672256829d6905837f5fab5"} Sep 08 11:45:09.336389 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:09.336026 2564 scope.go:117] "RemoveContainer" containerID="661b0752ad152ff9fb52a3f43a5fa3920072bc7e91a4d8ce9712180f5a91a15b" Sep 08 11:45:09.369031 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:09.368995 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h"] Sep 08 11:45:09.372836 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:09.372808 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a1721d-openshift-default-689f485974-7bh9h"] Sep 08 11:45:09.616178 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:09.616142 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v"] Sep 08 11:45:09.618721 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:45:09.618690 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a616290_eeb8_4957_adc5_233dbf6b6973.slice/crio-8ebe79ffdfa02cd9b7e140a30dcaf8b41c3efc177ff0cbe2f45dfccd1f4b07e6 WatchSource:0}: Error finding container 8ebe79ffdfa02cd9b7e140a30dcaf8b41c3efc177ff0cbe2f45dfccd1f4b07e6: Status 404 returned error can't find the container with id 8ebe79ffdfa02cd9b7e140a30dcaf8b41c3efc177ff0cbe2f45dfccd1f4b07e6 Sep 08 11:45:09.654140 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:09.654105 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77c83f90-b1bf-4342-ac82-a6d603d881ce" path="/var/lib/kubelet/pods/77c83f90-b1bf-4342-ac82-a6d603d881ce/volumes" Sep 08 11:45:10.343455 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:10.343356 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" event={"ID":"7a616290-eeb8-4957-adc5-233dbf6b6973","Type":"ContainerStarted","Data":"08d5458fe8366c4cffde844dcf841d7518893f5ba4b01fb4821a37b7cf205af5"} Sep 08 11:45:10.343455 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:10.343428 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" event={"ID":"7a616290-eeb8-4957-adc5-233dbf6b6973","Type":"ContainerStarted","Data":"8ebe79ffdfa02cd9b7e140a30dcaf8b41c3efc177ff0cbe2f45dfccd1f4b07e6"} Sep 08 11:45:10.343880 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:10.343565 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:10.363906 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:10.363858 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" podStartSLOduration=2.011595524 podStartE2EDuration="2.363843169s" totalImagesPullingTime="352.247645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:45:10.36281525 +0000 UTC m=+899.222002593" watchObservedRunningTime="2026-09-08 11:45:10.363843169 +0000 UTC m=+899.223030486" Sep 08 11:45:11.144273 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:11.144239 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:45:11.609343 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:11.609268 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 11:45:11.609343 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:11.609326 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 11:45:16.357675 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:16.357643 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:45:21.310943 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:21.310869 2564 patch_prober.go:28] interesting pod/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 08 11:45:21.310943 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:21.310927 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" podUID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 08 11:45:21.425667 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:21.425634 2564 generic.go:356] "Generic (PLEG): container finished" podID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerID="45190baa80187399eff43cf15069b9efa4c98449ab5ec5aeccdeae87e66427fd" exitCode=0 Sep 08 11:45:21.425834 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:21.425677 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" event={"ID":"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4","Type":"ContainerDied","Data":"45190baa80187399eff43cf15069b9efa4c98449ab5ec5aeccdeae87e66427fd"} Sep 08 11:45:21.426064 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:21.426051 2564 scope.go:117] "RemoveContainer" containerID="45190baa80187399eff43cf15069b9efa4c98449ab5ec5aeccdeae87e66427fd" Sep 08 11:45:21.427138 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:21.427108 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:45:21.427198 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:21.427177 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:45:21.427239 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:21.427211 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:45:22.434008 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:22.433974 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" event={"ID":"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4","Type":"ContainerStarted","Data":"f681cda9d7968e2473eea3d97d58d23a06568542b5da3fb74401aa56fc490dc4"} Sep 08 11:45:23.163722 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:23.163688 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:23.163904 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:23.163852 2564 patch_prober.go:28] interesting pod/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 08 11:45:23.163969 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:23.163897 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" podUID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 08 11:45:24.163764 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:24.163727 2564 patch_prober.go:28] interesting pod/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 08 11:45:24.164141 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:24.163783 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" podUID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 08 11:45:25.162887 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:25.162850 2564 patch_prober.go:28] interesting pod/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 08 11:45:25.163070 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:25.162903 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" podUID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 08 11:45:26.163478 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:26.163440 2564 patch_prober.go:28] interesting pod/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 08 11:45:26.163850 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:26.163500 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" podUID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 08 11:45:27.163662 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:27.163620 2564 patch_prober.go:28] interesting pod/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 08 11:45:27.164309 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:27.164265 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" podUID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 08 11:45:28.167556 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:28.167527 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:28.167958 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:28.167786 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:28.168395 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:28.168360 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:45:34.947924 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:34.947886 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:45:46.051153 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:45:46.051113 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:46:02.345037 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:02.345004 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:46:18.136512 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:18.136478 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:46:19.057437 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.056659 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v"] Sep 08 11:46:19.057437 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.057086 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" podUID="7a616290-eeb8-4957-adc5-233dbf6b6973" containerName="maas-api" containerID="cri-o://08d5458fe8366c4cffde844dcf841d7518893f5ba4b01fb4821a37b7cf205af5" gracePeriod=30 Sep 08 11:46:19.328342 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.328317 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:46:19.453138 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.453105 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49flk\" (UniqueName: \"kubernetes.io/projected/7a616290-eeb8-4957-adc5-233dbf6b6973-kube-api-access-49flk\") pod \"7a616290-eeb8-4957-adc5-233dbf6b6973\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " Sep 08 11:46:19.453321 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.453172 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a616290-eeb8-4957-adc5-233dbf6b6973-maas-api-tls\") pod \"7a616290-eeb8-4957-adc5-233dbf6b6973\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " Sep 08 11:46:19.453321 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.453194 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a616290-eeb8-4957-adc5-233dbf6b6973-openshift-service-ca\") pod \"7a616290-eeb8-4957-adc5-233dbf6b6973\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " Sep 08 11:46:19.453321 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.453216 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a616290-eeb8-4957-adc5-233dbf6b6973-metrics-certs\") pod \"7a616290-eeb8-4957-adc5-233dbf6b6973\" (UID: \"7a616290-eeb8-4957-adc5-233dbf6b6973\") " Sep 08 11:46:19.453631 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.453608 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a616290-eeb8-4957-adc5-233dbf6b6973-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a616290-eeb8-4957-adc5-233dbf6b6973" (UID: "7a616290-eeb8-4957-adc5-233dbf6b6973"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:46:19.455321 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.455297 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a616290-eeb8-4957-adc5-233dbf6b6973-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7a616290-eeb8-4957-adc5-233dbf6b6973" (UID: "7a616290-eeb8-4957-adc5-233dbf6b6973"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:46:19.455426 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.455349 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a616290-eeb8-4957-adc5-233dbf6b6973-kube-api-access-49flk" (OuterVolumeSpecName: "kube-api-access-49flk") pod "7a616290-eeb8-4957-adc5-233dbf6b6973" (UID: "7a616290-eeb8-4957-adc5-233dbf6b6973"). InnerVolumeSpecName "kube-api-access-49flk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:46:19.455426 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.455419 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a616290-eeb8-4957-adc5-233dbf6b6973-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a616290-eeb8-4957-adc5-233dbf6b6973" (UID: "7a616290-eeb8-4957-adc5-233dbf6b6973"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:46:19.554130 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.554096 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49flk\" (UniqueName: \"kubernetes.io/projected/7a616290-eeb8-4957-adc5-233dbf6b6973-kube-api-access-49flk\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:19.554130 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.554127 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a616290-eeb8-4957-adc5-233dbf6b6973-maas-api-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:19.554130 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.554136 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a616290-eeb8-4957-adc5-233dbf6b6973-openshift-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:19.554402 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.554146 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a616290-eeb8-4957-adc5-233dbf6b6973-metrics-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:19.763982 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.763952 2564 generic.go:356] "Generic (PLEG): container finished" podID="7a616290-eeb8-4957-adc5-233dbf6b6973" containerID="08d5458fe8366c4cffde844dcf841d7518893f5ba4b01fb4821a37b7cf205af5" exitCode=0 Sep 08 11:46:19.764137 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.763998 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" Sep 08 11:46:19.764137 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.764027 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" event={"ID":"7a616290-eeb8-4957-adc5-233dbf6b6973","Type":"ContainerDied","Data":"08d5458fe8366c4cffde844dcf841d7518893f5ba4b01fb4821a37b7cf205af5"} Sep 08 11:46:19.764137 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.764081 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v" event={"ID":"7a616290-eeb8-4957-adc5-233dbf6b6973","Type":"ContainerDied","Data":"8ebe79ffdfa02cd9b7e140a30dcaf8b41c3efc177ff0cbe2f45dfccd1f4b07e6"} Sep 08 11:46:19.764137 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.764105 2564 scope.go:117] "RemoveContainer" containerID="08d5458fe8366c4cffde844dcf841d7518893f5ba4b01fb4821a37b7cf205af5" Sep 08 11:46:19.773247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.773219 2564 scope.go:117] "RemoveContainer" containerID="08d5458fe8366c4cffde844dcf841d7518893f5ba4b01fb4821a37b7cf205af5" Sep 08 11:46:19.773564 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:46:19.773546 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d5458fe8366c4cffde844dcf841d7518893f5ba4b01fb4821a37b7cf205af5\": container with ID starting with 08d5458fe8366c4cffde844dcf841d7518893f5ba4b01fb4821a37b7cf205af5 not found: ID does not exist" containerID="08d5458fe8366c4cffde844dcf841d7518893f5ba4b01fb4821a37b7cf205af5" Sep 08 11:46:19.773615 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.773573 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d5458fe8366c4cffde844dcf841d7518893f5ba4b01fb4821a37b7cf205af5"} err="failed to get container status \"08d5458fe8366c4cffde844dcf841d7518893f5ba4b01fb4821a37b7cf205af5\": rpc error: code = NotFound desc = could not find container \"08d5458fe8366c4cffde844dcf841d7518893f5ba4b01fb4821a37b7cf205af5\": container with ID starting with 08d5458fe8366c4cffde844dcf841d7518893f5ba4b01fb4821a37b7cf205af5 not found: ID does not exist" Sep 08 11:46:19.786171 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.786147 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v"] Sep 08 11:46:19.791430 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:19.791409 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-20cf96-9497dd555-wbc6v"] Sep 08 11:46:21.654452 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:21.654419 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a616290-eeb8-4957-adc5-233dbf6b6973" path="/var/lib/kubelet/pods/7a616290-eeb8-4957-adc5-233dbf6b6973/volumes" Sep 08 11:46:27.624181 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:27.624147 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q"] Sep 08 11:46:27.624727 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:27.624420 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" podUID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerName="istio-proxy" containerID="cri-o://f681cda9d7968e2473eea3d97d58d23a06568542b5da3fb74401aa56fc490dc4" gracePeriod=30 Sep 08 11:46:28.170854 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:28.170814 2564 patch_prober.go:28] interesting pod/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 11:46:28.171018 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:28.170880 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" podUID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 11:46:30.150217 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:30.150184 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:46:32.841888 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.841855 2564 generic.go:356] "Generic (PLEG): container finished" podID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerID="f681cda9d7968e2473eea3d97d58d23a06568542b5da3fb74401aa56fc490dc4" exitCode=0 Sep 08 11:46:32.842334 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.841924 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" event={"ID":"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4","Type":"ContainerDied","Data":"f681cda9d7968e2473eea3d97d58d23a06568542b5da3fb74401aa56fc490dc4"} Sep 08 11:46:32.842334 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.841990 2564 scope.go:117] "RemoveContainer" containerID="45190baa80187399eff43cf15069b9efa4c98449ab5ec5aeccdeae87e66427fd" Sep 08 11:46:32.876631 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.876610 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:46:32.970991 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.970910 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-envoy\") pod \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " Sep 08 11:46:32.970991 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.970957 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-credential-socket\") pod \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " Sep 08 11:46:32.970991 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.970978 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-workload-certs\") pod \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " Sep 08 11:46:32.971244 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.971008 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-podinfo\") pod \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " Sep 08 11:46:32.971244 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.971077 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-data\") pod \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " Sep 08 11:46:32.971244 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.971116 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnmtp\" (UniqueName: \"kubernetes.io/projected/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-kube-api-access-pnmtp\") pod \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " Sep 08 11:46:32.971244 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.971149 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-workload-socket\") pod \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " Sep 08 11:46:32.971244 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.971191 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-token\") pod \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " Sep 08 11:46:32.971244 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.971214 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" (UID: "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:46:32.971244 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.971230 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" (UID: "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:46:32.971244 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.971219 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istiod-ca-cert\") pod \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\" (UID: \"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4\") " Sep 08 11:46:32.971688 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.971433 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-data" (OuterVolumeSpecName: "istio-data") pod "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" (UID: "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:46:32.971688 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.971609 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-data\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:32.971688 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.971620 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" (UID: "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:46:32.971688 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.971631 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-credential-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:32.971688 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.971670 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-workload-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:32.971688 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.971628 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" (UID: "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:46:32.973216 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.973167 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" (UID: "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:46:32.973322 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.973249 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-kube-api-access-pnmtp" (OuterVolumeSpecName: "kube-api-access-pnmtp") pod "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" (UID: "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4"). InnerVolumeSpecName "kube-api-access-pnmtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:46:32.973322 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.973260 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" (UID: "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 11:46:32.973546 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:32.973521 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-token" (OuterVolumeSpecName: "istio-token") pod "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" (UID: "5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:46:33.073101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:33.073075 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnmtp\" (UniqueName: \"kubernetes.io/projected/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-kube-api-access-pnmtp\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:33.073101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:33.073100 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-workload-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:33.073262 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:33.073110 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:33.073262 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:33.073119 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istiod-ca-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:33.073262 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:33.073127 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-envoy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:33.073262 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:33.073134 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4-istio-podinfo\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:33.848992 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:33.848966 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" Sep 08 11:46:33.849446 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:33.849014 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q" event={"ID":"5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4","Type":"ContainerDied","Data":"a7fa3dcefce3c25d6b6cff4ab22ff3694ad4f8a1235262bd99a5cd391706a7a4"} Sep 08 11:46:33.849446 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:33.849072 2564 scope.go:117] "RemoveContainer" containerID="f681cda9d7968e2473eea3d97d58d23a06568542b5da3fb74401aa56fc490dc4" Sep 08 11:46:33.871614 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:33.871591 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q"] Sep 08 11:46:33.876693 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:33.876671 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-20cf96-openshift-default-77b4f5d8b8-9f96q"] Sep 08 11:46:35.653278 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:35.653243 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" path="/var/lib/kubelet/pods/5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4/volumes" Sep 08 11:46:36.963915 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:36.963883 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-95669b-29814465-xbbz9"] Sep 08 11:46:36.968832 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:36.968805 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-95669b-29814465-xbbz9"] Sep 08 11:46:37.653453 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:37.653420 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0ef437b-5d04-48bf-b696-1bd6218d5571" path="/var/lib/kubelet/pods/e0ef437b-5d04-48bf-b696-1bd6218d5571/volumes" Sep 08 11:46:39.343536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:39.343497 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:46:55.818545 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:55.818509 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l"] Sep 08 11:46:55.818990 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:55.818735 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" podUID="404a4d9c-92d4-4e9e-acbb-ce089d0f4196" containerName="maas-api" containerID="cri-o://fc4852fb12a5baa6a8e8990d3d3b319002c8507899eb0a17ca09fd2fbfd45426" gracePeriod=30 Sep 08 11:46:55.995622 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:55.995584 2564 generic.go:356] "Generic (PLEG): container finished" podID="404a4d9c-92d4-4e9e-acbb-ce089d0f4196" containerID="fc4852fb12a5baa6a8e8990d3d3b319002c8507899eb0a17ca09fd2fbfd45426" exitCode=0 Sep 08 11:46:55.995764 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:55.995690 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" event={"ID":"404a4d9c-92d4-4e9e-acbb-ce089d0f4196","Type":"ContainerDied","Data":"fc4852fb12a5baa6a8e8990d3d3b319002c8507899eb0a17ca09fd2fbfd45426"} Sep 08 11:46:56.078345 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:56.078289 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:46:56.187983 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:56.187955 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-openshift-service-ca\") pod \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " Sep 08 11:46:56.188151 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:56.188067 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-metrics-certs\") pod \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " Sep 08 11:46:56.188151 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:56.188127 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vntgw\" (UniqueName: \"kubernetes.io/projected/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-kube-api-access-vntgw\") pod \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " Sep 08 11:46:56.188151 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:56.188148 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-maas-api-tls\") pod \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\" (UID: \"404a4d9c-92d4-4e9e-acbb-ce089d0f4196\") " Sep 08 11:46:56.188472 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:56.188430 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "404a4d9c-92d4-4e9e-acbb-ce089d0f4196" (UID: "404a4d9c-92d4-4e9e-acbb-ce089d0f4196"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:46:56.190320 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:56.190289 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "404a4d9c-92d4-4e9e-acbb-ce089d0f4196" (UID: "404a4d9c-92d4-4e9e-acbb-ce089d0f4196"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:46:56.190712 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:56.190687 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "404a4d9c-92d4-4e9e-acbb-ce089d0f4196" (UID: "404a4d9c-92d4-4e9e-acbb-ce089d0f4196"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:46:56.190775 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:56.190740 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-kube-api-access-vntgw" (OuterVolumeSpecName: "kube-api-access-vntgw") pod "404a4d9c-92d4-4e9e-acbb-ce089d0f4196" (UID: "404a4d9c-92d4-4e9e-acbb-ce089d0f4196"). InnerVolumeSpecName "kube-api-access-vntgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:46:56.289621 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:56.289583 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-metrics-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:56.289621 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:56.289614 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vntgw\" (UniqueName: \"kubernetes.io/projected/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-kube-api-access-vntgw\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:56.289621 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:56.289625 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-maas-api-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:56.289847 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:56.289634 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/404a4d9c-92d4-4e9e-acbb-ce089d0f4196-openshift-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:46:57.003327 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:57.003300 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" Sep 08 11:46:57.003327 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:57.003321 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l" event={"ID":"404a4d9c-92d4-4e9e-acbb-ce089d0f4196","Type":"ContainerDied","Data":"32a3ee13a1c5912b1a76a8c05c3f4bfe5c41a3e3bd2785ace37a1713dad5610d"} Sep 08 11:46:57.003819 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:57.003392 2564 scope.go:117] "RemoveContainer" containerID="fc4852fb12a5baa6a8e8990d3d3b319002c8507899eb0a17ca09fd2fbfd45426" Sep 08 11:46:57.028516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:57.028489 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l"] Sep 08 11:46:57.035696 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:57.035672 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-425b2c-b4949fdf7-54g6l"] Sep 08 11:46:57.653350 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:57.653318 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="404a4d9c-92d4-4e9e-acbb-ce089d0f4196" path="/var/lib/kubelet/pods/404a4d9c-92d4-4e9e-acbb-ce089d0f4196/volumes" Sep 08 11:46:58.438185 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:46:58.438141 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:47:03.744035 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:03.743999 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l"] Sep 08 11:47:03.744571 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:03.744203 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" podUID="6953953d-ae92-48a2-8326-841c7d40a417" containerName="istio-proxy" containerID="cri-o://87f80cab32858a7e01d11612d5a9f41658c5d97593516d71a25b8425bce4bed7" gracePeriod=30 Sep 08 11:47:04.568438 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.568403 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc"] Sep 08 11:47:04.569084 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.569056 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerName="istio-proxy" Sep 08 11:47:04.569084 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.569084 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerName="istio-proxy" Sep 08 11:47:04.569257 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.569097 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="404a4d9c-92d4-4e9e-acbb-ce089d0f4196" containerName="maas-api" Sep 08 11:47:04.569257 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.569105 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="404a4d9c-92d4-4e9e-acbb-ce089d0f4196" containerName="maas-api" Sep 08 11:47:04.569257 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.569122 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77c83f90-b1bf-4342-ac82-a6d603d881ce" containerName="istio-proxy" Sep 08 11:47:04.569257 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.569130 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="77c83f90-b1bf-4342-ac82-a6d603d881ce" containerName="istio-proxy" Sep 08 11:47:04.569257 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.569141 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a616290-eeb8-4957-adc5-233dbf6b6973" containerName="maas-api" Sep 08 11:47:04.569257 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.569148 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a616290-eeb8-4957-adc5-233dbf6b6973" containerName="maas-api" Sep 08 11:47:04.569257 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.569158 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerName="istio-proxy" Sep 08 11:47:04.569257 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.569166 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerName="istio-proxy" Sep 08 11:47:04.569665 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.569274 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="77c83f90-b1bf-4342-ac82-a6d603d881ce" containerName="istio-proxy" Sep 08 11:47:04.569665 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.569289 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerName="istio-proxy" Sep 08 11:47:04.569665 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.569302 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a616290-eeb8-4957-adc5-233dbf6b6973" containerName="maas-api" Sep 08 11:47:04.569665 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.569316 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="404a4d9c-92d4-4e9e-acbb-ce089d0f4196" containerName="maas-api" Sep 08 11:47:04.574120 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.574098 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.576488 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.576467 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-838a68-openshift-default-dockercfg-fbzvs\"" Sep 08 11:47:04.586122 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.586099 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc"] Sep 08 11:47:04.667670 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.667630 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-credential-socket\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.667841 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.667678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-podinfo\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.667841 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.667713 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-envoy\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.667841 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.667750 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-data\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.667841 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.667773 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-token\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.667999 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.667881 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-workload-certs\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.667999 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.667950 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-workload-socket\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.668099 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.668039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjpqz\" (UniqueName: \"kubernetes.io/projected/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-kube-api-access-xjpqz\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.668099 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.668085 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istiod-ca-cert\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.769683 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.769644 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istiod-ca-cert\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.769683 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.769689 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-credential-socket\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.770192 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.769714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-podinfo\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.770192 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.769742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-envoy\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.770192 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.769783 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-data\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.770192 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.769817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-token\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.770192 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.769971 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-workload-certs\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.770192 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.770053 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-workload-socket\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.770192 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.770110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjpqz\" (UniqueName: \"kubernetes.io/projected/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-kube-api-access-xjpqz\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.770192 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.770136 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-credential-socket\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.770666 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.770304 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-workload-certs\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.770666 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.770399 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-workload-socket\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.770666 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.770402 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istiod-ca-cert\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.770666 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.770595 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-data\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.772180 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.772152 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-envoy\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.772508 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.772489 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-podinfo\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.779397 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.779343 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-token\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.779670 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.779649 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjpqz\" (UniqueName: \"kubernetes.io/projected/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-kube-api-access-xjpqz\") pod \"e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:04.886238 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:04.886211 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:05.166594 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:05.166561 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc"] Sep 08 11:47:05.167573 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:47:05.167544 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c99d63f_464e_48f0_b9c6_3893bb4c96ee.slice/crio-833a8688d3455ce58348584d99a03f79b5d406fdaea344881211c53e2a85d6af WatchSource:0}: Error finding container 833a8688d3455ce58348584d99a03f79b5d406fdaea344881211c53e2a85d6af: Status 404 returned error can't find the container with id 833a8688d3455ce58348584d99a03f79b5d406fdaea344881211c53e2a85d6af Sep 08 11:47:05.170021 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:05.169990 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:05.170104 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:05.170055 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:05.170104 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:05.170082 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:06.058943 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:06.058907 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" event={"ID":"0c99d63f-464e-48f0-b9c6-3893bb4c96ee","Type":"ContainerStarted","Data":"dbb4ad2f6c4a44e7b343fa9925e57f7faef22bd9991060d3d307b2abbccfa9e9"} Sep 08 11:47:06.059339 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:06.058961 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" event={"ID":"0c99d63f-464e-48f0-b9c6-3893bb4c96ee","Type":"ContainerStarted","Data":"833a8688d3455ce58348584d99a03f79b5d406fdaea344881211c53e2a85d6af"} Sep 08 11:47:06.084040 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:06.083983 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" podStartSLOduration=2.08396235 podStartE2EDuration="2.08396235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:47:06.081459052 +0000 UTC m=+1014.940646418" watchObservedRunningTime="2026-09-08 11:47:06.08396235 +0000 UTC m=+1014.943149671" Sep 08 11:47:06.886844 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:06.886804 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:06.892072 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:06.892046 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:07.065119 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:07.065084 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:07.066024 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:07.066003 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:08.457192 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:08.457157 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:47:09.003233 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.003203 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:47:09.003809 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.003785 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6953953d-ae92-48a2-8326-841c7d40a417-istio-podinfo\") pod \"6953953d-ae92-48a2-8326-841c7d40a417\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " Sep 08 11:47:09.003874 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.003829 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-workload-socket\") pod \"6953953d-ae92-48a2-8326-841c7d40a417\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " Sep 08 11:47:09.003874 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.003858 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-istio-envoy\") pod \"6953953d-ae92-48a2-8326-841c7d40a417\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " Sep 08 11:47:09.003986 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.003895 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6953953d-ae92-48a2-8326-841c7d40a417-istio-token\") pod \"6953953d-ae92-48a2-8326-841c7d40a417\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " Sep 08 11:47:09.003986 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.003940 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6953953d-ae92-48a2-8326-841c7d40a417-istiod-ca-cert\") pod \"6953953d-ae92-48a2-8326-841c7d40a417\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " Sep 08 11:47:09.004094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.003983 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nghjz\" (UniqueName: \"kubernetes.io/projected/6953953d-ae92-48a2-8326-841c7d40a417-kube-api-access-nghjz\") pod \"6953953d-ae92-48a2-8326-841c7d40a417\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " Sep 08 11:47:09.004094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.004013 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-credential-socket\") pod \"6953953d-ae92-48a2-8326-841c7d40a417\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " Sep 08 11:47:09.004094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.004063 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-workload-certs\") pod \"6953953d-ae92-48a2-8326-841c7d40a417\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " Sep 08 11:47:09.004094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.004088 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-istio-data\") pod \"6953953d-ae92-48a2-8326-841c7d40a417\" (UID: \"6953953d-ae92-48a2-8326-841c7d40a417\") " Sep 08 11:47:09.004564 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.004456 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6953953d-ae92-48a2-8326-841c7d40a417" (UID: "6953953d-ae92-48a2-8326-841c7d40a417"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:47:09.004564 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.004532 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6953953d-ae92-48a2-8326-841c7d40a417" (UID: "6953953d-ae92-48a2-8326-841c7d40a417"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:47:09.004564 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.004549 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-istio-data" (OuterVolumeSpecName: "istio-data") pod "6953953d-ae92-48a2-8326-841c7d40a417" (UID: "6953953d-ae92-48a2-8326-841c7d40a417"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:47:09.005124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.005099 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6953953d-ae92-48a2-8326-841c7d40a417-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6953953d-ae92-48a2-8326-841c7d40a417" (UID: "6953953d-ae92-48a2-8326-841c7d40a417"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:47:09.005300 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.005279 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6953953d-ae92-48a2-8326-841c7d40a417" (UID: "6953953d-ae92-48a2-8326-841c7d40a417"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:47:09.006511 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.006440 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6953953d-ae92-48a2-8326-841c7d40a417" (UID: "6953953d-ae92-48a2-8326-841c7d40a417"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:47:09.006511 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.006484 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6953953d-ae92-48a2-8326-841c7d40a417-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6953953d-ae92-48a2-8326-841c7d40a417" (UID: "6953953d-ae92-48a2-8326-841c7d40a417"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 11:47:09.006671 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.006591 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6953953d-ae92-48a2-8326-841c7d40a417-istio-token" (OuterVolumeSpecName: "istio-token") pod "6953953d-ae92-48a2-8326-841c7d40a417" (UID: "6953953d-ae92-48a2-8326-841c7d40a417"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:47:09.006968 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.006947 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6953953d-ae92-48a2-8326-841c7d40a417-kube-api-access-nghjz" (OuterVolumeSpecName: "kube-api-access-nghjz") pod "6953953d-ae92-48a2-8326-841c7d40a417" (UID: "6953953d-ae92-48a2-8326-841c7d40a417"). InnerVolumeSpecName "kube-api-access-nghjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:47:09.077991 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.077951 2564 generic.go:356] "Generic (PLEG): container finished" podID="6953953d-ae92-48a2-8326-841c7d40a417" containerID="87f80cab32858a7e01d11612d5a9f41658c5d97593516d71a25b8425bce4bed7" exitCode=0 Sep 08 11:47:09.078130 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.078001 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" Sep 08 11:47:09.078130 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.078005 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" event={"ID":"6953953d-ae92-48a2-8326-841c7d40a417","Type":"ContainerDied","Data":"87f80cab32858a7e01d11612d5a9f41658c5d97593516d71a25b8425bce4bed7"} Sep 08 11:47:09.078130 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.078066 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l" event={"ID":"6953953d-ae92-48a2-8326-841c7d40a417","Type":"ContainerDied","Data":"e0526e3730a5d07f8e984f0498d6ff145fc47e8b42c77d8840e89cb5f065d938"} Sep 08 11:47:09.078130 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.078098 2564 scope.go:117] "RemoveContainer" containerID="87f80cab32858a7e01d11612d5a9f41658c5d97593516d71a25b8425bce4bed7" Sep 08 11:47:09.088447 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.088428 2564 scope.go:117] "RemoveContainer" containerID="d4d08116e5a2a6177679bdc1eeab1cb8900c456ac49b8138c07bb1110ce8fcfc" Sep 08 11:47:09.097162 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.097138 2564 scope.go:117] "RemoveContainer" containerID="87f80cab32858a7e01d11612d5a9f41658c5d97593516d71a25b8425bce4bed7" Sep 08 11:47:09.097477 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:47:09.097449 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f80cab32858a7e01d11612d5a9f41658c5d97593516d71a25b8425bce4bed7\": container with ID starting with 87f80cab32858a7e01d11612d5a9f41658c5d97593516d71a25b8425bce4bed7 not found: ID does not exist" containerID="87f80cab32858a7e01d11612d5a9f41658c5d97593516d71a25b8425bce4bed7" Sep 08 11:47:09.097586 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.097483 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f80cab32858a7e01d11612d5a9f41658c5d97593516d71a25b8425bce4bed7"} err="failed to get container status \"87f80cab32858a7e01d11612d5a9f41658c5d97593516d71a25b8425bce4bed7\": rpc error: code = NotFound desc = could not find container \"87f80cab32858a7e01d11612d5a9f41658c5d97593516d71a25b8425bce4bed7\": container with ID starting with 87f80cab32858a7e01d11612d5a9f41658c5d97593516d71a25b8425bce4bed7 not found: ID does not exist" Sep 08 11:47:09.097586 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.097501 2564 scope.go:117] "RemoveContainer" containerID="d4d08116e5a2a6177679bdc1eeab1cb8900c456ac49b8138c07bb1110ce8fcfc" Sep 08 11:47:09.097813 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:47:09.097788 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d08116e5a2a6177679bdc1eeab1cb8900c456ac49b8138c07bb1110ce8fcfc\": container with ID starting with d4d08116e5a2a6177679bdc1eeab1cb8900c456ac49b8138c07bb1110ce8fcfc not found: ID does not exist" containerID="d4d08116e5a2a6177679bdc1eeab1cb8900c456ac49b8138c07bb1110ce8fcfc" Sep 08 11:47:09.097907 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.097812 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d08116e5a2a6177679bdc1eeab1cb8900c456ac49b8138c07bb1110ce8fcfc"} err="failed to get container status \"d4d08116e5a2a6177679bdc1eeab1cb8900c456ac49b8138c07bb1110ce8fcfc\": rpc error: code = NotFound desc = could not find container \"d4d08116e5a2a6177679bdc1eeab1cb8900c456ac49b8138c07bb1110ce8fcfc\": container with ID starting with d4d08116e5a2a6177679bdc1eeab1cb8900c456ac49b8138c07bb1110ce8fcfc not found: ID does not exist" Sep 08 11:47:09.104963 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.104940 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-workload-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:47:09.104963 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.104962 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-istio-envoy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:47:09.105125 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.104971 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6953953d-ae92-48a2-8326-841c7d40a417-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:47:09.105125 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.104980 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6953953d-ae92-48a2-8326-841c7d40a417-istiod-ca-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:47:09.105125 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.104988 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nghjz\" (UniqueName: \"kubernetes.io/projected/6953953d-ae92-48a2-8326-841c7d40a417-kube-api-access-nghjz\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:47:09.105125 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.105001 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-credential-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:47:09.105125 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.105013 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-workload-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:47:09.105125 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.105022 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6953953d-ae92-48a2-8326-841c7d40a417-istio-data\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:47:09.105125 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.105029 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6953953d-ae92-48a2-8326-841c7d40a417-istio-podinfo\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:47:09.122213 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.122190 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l"] Sep 08 11:47:09.134343 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.134313 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-425b2c-openshift-default-658cc4654d-9m68l"] Sep 08 11:47:09.653092 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:09.653059 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6953953d-ae92-48a2-8326-841c7d40a417" path="/var/lib/kubelet/pods/6953953d-ae92-48a2-8326-841c7d40a417/volumes" Sep 08 11:47:16.620815 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.620772 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm"] Sep 08 11:47:16.621294 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.621273 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6953953d-ae92-48a2-8326-841c7d40a417" containerName="istio-proxy" Sep 08 11:47:16.621294 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.621289 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6953953d-ae92-48a2-8326-841c7d40a417" containerName="istio-proxy" Sep 08 11:47:16.621483 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.621299 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6953953d-ae92-48a2-8326-841c7d40a417" containerName="istio-proxy" Sep 08 11:47:16.621483 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.621305 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6953953d-ae92-48a2-8326-841c7d40a417" containerName="istio-proxy" Sep 08 11:47:16.621483 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.621362 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fcb63a2-0dd6-4a6e-b98b-f5200f947ca4" containerName="istio-proxy" Sep 08 11:47:16.621483 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.621397 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6953953d-ae92-48a2-8326-841c7d40a417" containerName="istio-proxy" Sep 08 11:47:16.621483 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.621404 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6953953d-ae92-48a2-8326-841c7d40a417" containerName="istio-proxy" Sep 08 11:47:16.626402 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.626380 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.629108 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.629076 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2fe44c90-openshift-default-dockercfg-5gwcf\"" Sep 08 11:47:16.637628 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.637603 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm"] Sep 08 11:47:16.666595 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.666565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istiod-ca-cert\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.666732 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.666604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-825wc\" (UniqueName: \"kubernetes.io/projected/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-kube-api-access-825wc\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.666732 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.666648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-data\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.666732 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.666699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-token\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.666852 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.666739 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-credential-socket\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.666852 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.666790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-workload-certs\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.666852 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.666829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-envoy\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.667010 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.666897 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-workload-socket\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.667010 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.666919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-podinfo\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.767864 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.767828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istiod-ca-cert\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.767864 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.767866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-825wc\" (UniqueName: \"kubernetes.io/projected/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-kube-api-access-825wc\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.768057 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.767899 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-data\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.768057 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.767931 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-token\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.768057 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.767964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-credential-socket\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.768057 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.768022 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-workload-certs\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.768285 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.768059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-envoy\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.768285 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.768108 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-workload-socket\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.768285 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.768139 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-podinfo\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.768464 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.768387 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-credential-socket\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.768464 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.768424 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-data\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.768533 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.768467 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-workload-certs\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.768616 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.768588 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-workload-socket\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.768733 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.768653 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istiod-ca-cert\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.770465 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.770439 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-envoy\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.770568 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.770552 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-podinfo\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.778622 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.778574 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-token\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.779268 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.779245 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-825wc\" (UniqueName: \"kubernetes.io/projected/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-kube-api-access-825wc\") pod \"e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:16.938747 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:16.938709 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:17.246970 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:17.246939 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm"] Sep 08 11:47:17.248593 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:47:17.248570 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cab4e58_85f1_4e71_bc6a_87a67461bdd1.slice/crio-76ee93029e48d0fdb89fed5aa02297d83bbbb99120ee69a06135a33c74177cf4 WatchSource:0}: Error finding container 76ee93029e48d0fdb89fed5aa02297d83bbbb99120ee69a06135a33c74177cf4: Status 404 returned error can't find the container with id 76ee93029e48d0fdb89fed5aa02297d83bbbb99120ee69a06135a33c74177cf4 Sep 08 11:47:17.250884 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:17.250849 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:17.250974 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:17.250918 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:17.250974 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:17.250949 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:18.138440 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:18.138406 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" event={"ID":"2cab4e58-85f1-4e71-bc6a-87a67461bdd1","Type":"ContainerStarted","Data":"2bfcf014058d47d9279225331e4a89391e109a4d68fb1887fc913ae16bdc2715"} Sep 08 11:47:18.138817 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:18.138458 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" event={"ID":"2cab4e58-85f1-4e71-bc6a-87a67461bdd1","Type":"ContainerStarted","Data":"76ee93029e48d0fdb89fed5aa02297d83bbbb99120ee69a06135a33c74177cf4"} Sep 08 11:47:18.172031 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:18.171968 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" podStartSLOduration=2.171950903 podStartE2EDuration="2.171950903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:47:18.169530672 +0000 UTC m=+1027.028718038" watchObservedRunningTime="2026-09-08 11:47:18.171950903 +0000 UTC m=+1027.031138220" Sep 08 11:47:18.939721 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:18.939674 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:18.944939 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:18.944912 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:19.144619 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:19.144584 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:19.145579 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:19.145560 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:19.864913 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:19.864874 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:47:21.405825 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.405794 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r"] Sep 08 11:47:21.409303 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.409283 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:21.412101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.412077 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 11:47:21.412245 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.412128 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2fe44c90-serving-cert\"" Sep 08 11:47:21.412245 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.412182 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 08 11:47:21.413573 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.413552 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b5t6p\"" Sep 08 11:47:21.413573 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.413550 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 08 11:47:21.431449 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.429089 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r"] Sep 08 11:47:21.516562 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.516527 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwhcc\" (UniqueName: \"kubernetes.io/projected/158fca76-814c-4abc-95a8-dce964ef4264-kube-api-access-cwhcc\") pod \"maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:21.516776 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.516587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/158fca76-814c-4abc-95a8-dce964ef4264-openshift-service-ca\") pod \"maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:21.516776 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.516631 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/158fca76-814c-4abc-95a8-dce964ef4264-maas-api-tls\") pod \"maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:21.516776 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.516739 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158fca76-814c-4abc-95a8-dce964ef4264-metrics-certs\") pod \"maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:21.617552 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.617516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhcc\" (UniqueName: \"kubernetes.io/projected/158fca76-814c-4abc-95a8-dce964ef4264-kube-api-access-cwhcc\") pod \"maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:21.617749 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.617579 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/158fca76-814c-4abc-95a8-dce964ef4264-openshift-service-ca\") pod \"maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:21.617749 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.617611 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/158fca76-814c-4abc-95a8-dce964ef4264-maas-api-tls\") pod \"maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:21.617749 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.617705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158fca76-814c-4abc-95a8-dce964ef4264-metrics-certs\") pod \"maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:21.617921 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:47:21.617793 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-serving-cert: secret "maas-api-e2e-mt-2fe44c90-serving-cert" not found Sep 08 11:47:21.617921 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:47:21.617905 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/158fca76-814c-4abc-95a8-dce964ef4264-maas-api-tls podName:158fca76-814c-4abc-95a8-dce964ef4264 nodeName:}" failed. No retries permitted until 2026-09-08 11:47:22.117883488 +0000 UTC m=+1030.977070798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/158fca76-814c-4abc-95a8-dce964ef4264-maas-api-tls") pod "maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" (UID: "158fca76-814c-4abc-95a8-dce964ef4264") : secret "maas-api-e2e-mt-2fe44c90-serving-cert" not found Sep 08 11:47:21.618315 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.618289 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/158fca76-814c-4abc-95a8-dce964ef4264-openshift-service-ca\") pod \"maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:21.620118 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.620099 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158fca76-814c-4abc-95a8-dce964ef4264-metrics-certs\") pod \"maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:21.634459 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:21.634433 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwhcc\" (UniqueName: \"kubernetes.io/projected/158fca76-814c-4abc-95a8-dce964ef4264-kube-api-access-cwhcc\") pod \"maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:22.122953 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:22.122906 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/158fca76-814c-4abc-95a8-dce964ef4264-maas-api-tls\") pod \"maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:22.125474 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:22.125447 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/158fca76-814c-4abc-95a8-dce964ef4264-maas-api-tls\") pod \"maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:22.333320 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:22.333276 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:22.612430 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:22.612398 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r"] Sep 08 11:47:22.614360 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:47:22.614317 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158fca76_814c_4abc_95a8_dce964ef4264.slice/crio-2d8c22dc20b996a39f005713543c9a2b9ddc4f997ea4842e26e40510dc65152e WatchSource:0}: Error finding container 2d8c22dc20b996a39f005713543c9a2b9ddc4f997ea4842e26e40510dc65152e: Status 404 returned error can't find the container with id 2d8c22dc20b996a39f005713543c9a2b9ddc4f997ea4842e26e40510dc65152e Sep 08 11:47:22.616045 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:22.616024 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 11:47:22.892467 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:22.892418 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": read tcp 10.133.0.2:50180->10.133.0.60:15021: read: connection reset by peer" start-of-body= Sep 08 11:47:22.892660 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:22.892494 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" podUID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": read tcp 10.133.0.2:50180->10.133.0.60:15021: read: connection reset by peer" Sep 08 11:47:23.168785 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:23.168702 2564 generic.go:356] "Generic (PLEG): container finished" podID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" containerID="dbb4ad2f6c4a44e7b343fa9925e57f7faef22bd9991060d3d307b2abbccfa9e9" exitCode=0 Sep 08 11:47:23.168981 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:23.168779 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" event={"ID":"0c99d63f-464e-48f0-b9c6-3893bb4c96ee","Type":"ContainerDied","Data":"dbb4ad2f6c4a44e7b343fa9925e57f7faef22bd9991060d3d307b2abbccfa9e9"} Sep 08 11:47:23.169258 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:23.169232 2564 scope.go:117] "RemoveContainer" containerID="dbb4ad2f6c4a44e7b343fa9925e57f7faef22bd9991060d3d307b2abbccfa9e9" Sep 08 11:47:23.172995 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:23.172957 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:23.173108 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:23.173080 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:23.173175 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:23.173140 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:23.173701 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:23.173672 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" event={"ID":"158fca76-814c-4abc-95a8-dce964ef4264","Type":"ContainerStarted","Data":"ed50a11882d84498194c35a6a8f026baeaff6e64126dfb91a3ff921f45a36e42"} Sep 08 11:47:23.173781 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:23.173714 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" event={"ID":"158fca76-814c-4abc-95a8-dce964ef4264","Type":"ContainerStarted","Data":"2d8c22dc20b996a39f005713543c9a2b9ddc4f997ea4842e26e40510dc65152e"} Sep 08 11:47:23.173781 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:23.173742 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:23.217523 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:23.217476 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" podStartSLOduration=1.8094955910000001 podStartE2EDuration="2.217459375s" totalImagesPullingTime="407.963784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:47:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:47:23.214561343 +0000 UTC m=+1032.073748675" watchObservedRunningTime="2026-09-08 11:47:23.217459375 +0000 UTC m=+1032.076646694" Sep 08 11:47:24.186645 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:24.186605 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" event={"ID":"0c99d63f-464e-48f0-b9c6-3893bb4c96ee","Type":"ContainerStarted","Data":"d3ca3672b02afe91f8e6b2227ce61737916ec0fe4bbd8fdc7bfcd6af618cbc39"} Sep 08 11:47:24.887304 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:24.887271 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:24.887667 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:24.887644 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 08 11:47:24.887826 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:24.887801 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" podUID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 08 11:47:25.888051 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:25.888003 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 08 11:47:25.888575 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:25.888073 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" podUID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 08 11:47:26.886866 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:26.886824 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 08 11:47:26.887178 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:26.887149 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" podUID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 08 11:47:27.887634 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:27.887588 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 08 11:47:27.888143 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:27.887660 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" podUID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 08 11:47:28.887587 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:28.887530 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 08 11:47:28.887937 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:28.887881 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" podUID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 08 11:47:29.201069 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:29.200970 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:47:29.893915 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:29.893878 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:29.894412 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:29.894332 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:29.900781 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:29.900750 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:47:33.509378 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.509322 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv"] Sep 08 11:47:33.512867 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.512831 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.516603 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.516576 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-bb9b64-openshift-default-dockercfg-v77r4\"" Sep 08 11:47:33.526361 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.526339 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv"] Sep 08 11:47:33.645082 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.645046 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-credential-socket\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.645248 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.645108 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-workload-socket\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.645248 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.645166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-envoy\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.645248 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.645232 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istiod-ca-cert\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.645361 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.645307 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-workload-certs\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.645361 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.645327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-podinfo\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.645482 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.645415 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-token\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.645482 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.645446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-data\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.645482 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.645464 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grl2m\" (UniqueName: \"kubernetes.io/projected/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-kube-api-access-grl2m\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.746621 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.746582 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-token\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.746819 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.746630 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-data\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.746819 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.746648 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grl2m\" (UniqueName: \"kubernetes.io/projected/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-kube-api-access-grl2m\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.746819 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.746683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-credential-socket\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.746819 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.746720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-workload-socket\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.746819 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.746749 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-envoy\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.746819 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.746772 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istiod-ca-cert\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.747154 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.746836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-workload-certs\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.747154 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.746865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-podinfo\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.747154 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.747062 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-data\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.747448 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.747428 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istiod-ca-cert\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.747711 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.747692 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-credential-socket\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.747784 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.747734 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-workload-socket\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.747842 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.747785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-workload-certs\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.749349 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.749329 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-podinfo\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.749484 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.749467 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-envoy\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.756029 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.756007 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-token\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.756358 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.756340 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grl2m\" (UniqueName: \"kubernetes.io/projected/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-kube-api-access-grl2m\") pod \"e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:33.827015 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:33.826909 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:34.119795 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:34.119767 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv"] Sep 08 11:47:34.120944 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:47:34.120918 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3ef434_9302_44ca_86aa_eeb8c8a9c3ff.slice/crio-1e1fe6815df79348f2d9379ddef5eb878ed48da0ac5da907ab40e8405bf1a15e WatchSource:0}: Error finding container 1e1fe6815df79348f2d9379ddef5eb878ed48da0ac5da907ab40e8405bf1a15e: Status 404 returned error can't find the container with id 1e1fe6815df79348f2d9379ddef5eb878ed48da0ac5da907ab40e8405bf1a15e Sep 08 11:47:34.123305 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:34.123178 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:34.123305 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:34.123242 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:34.123305 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:34.123269 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:34.263343 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:34.263306 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" event={"ID":"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff","Type":"ContainerStarted","Data":"8e0d6905b20f1f7decc23309293ac2b4a027b8eca0dda96653129498384de662"} Sep 08 11:47:34.263546 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:34.263378 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" event={"ID":"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff","Type":"ContainerStarted","Data":"1e1fe6815df79348f2d9379ddef5eb878ed48da0ac5da907ab40e8405bf1a15e"} Sep 08 11:47:34.285723 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:34.285642 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" podStartSLOduration=1.285626683 podStartE2EDuration="1.285626683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:47:34.28495021 +0000 UTC m=+1043.144137734" watchObservedRunningTime="2026-09-08 11:47:34.285626683 +0000 UTC m=+1043.144814001" Sep 08 11:47:34.827860 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:34.827778 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:35.140889 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:35.140854 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:47:35.832730 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:35.832701 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:36.278296 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:36.278260 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:36.279408 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:36.279387 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:38.293128 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:38.293094 2564 generic.go:356] "Generic (PLEG): container finished" podID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" containerID="2bfcf014058d47d9279225331e4a89391e109a4d68fb1887fc913ae16bdc2715" exitCode=0 Sep 08 11:47:38.293648 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:38.293169 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" event={"ID":"2cab4e58-85f1-4e71-bc6a-87a67461bdd1","Type":"ContainerDied","Data":"2bfcf014058d47d9279225331e4a89391e109a4d68fb1887fc913ae16bdc2715"} Sep 08 11:47:38.293792 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:38.293774 2564 scope.go:117] "RemoveContainer" containerID="2bfcf014058d47d9279225331e4a89391e109a4d68fb1887fc913ae16bdc2715" Sep 08 11:47:38.294788 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:38.294756 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:38.294882 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:38.294823 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:38.294882 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:38.294851 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:39.305138 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:39.305100 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" event={"ID":"2cab4e58-85f1-4e71-bc6a-87a67461bdd1","Type":"ContainerStarted","Data":"b4de367c300d463874152cfdd0d493bb80743f191b0e05a402626d446240a592"} Sep 08 11:47:39.939438 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:39.939400 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:39.939634 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:39.939609 2564 patch_prober.go:28] interesting pod/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 08 11:47:39.939692 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:39.939674 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" podUID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 08 11:47:40.939195 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:40.939161 2564 patch_prober.go:28] interesting pod/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 08 11:47:40.939573 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:40.939220 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" podUID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 08 11:47:41.939316 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:41.939280 2564 patch_prober.go:28] interesting pod/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 08 11:47:41.939719 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:41.939338 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" podUID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 08 11:47:42.939874 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:42.939838 2564 patch_prober.go:28] interesting pod/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 08 11:47:42.940251 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:42.939890 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" podUID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 08 11:47:43.939863 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:43.939826 2564 patch_prober.go:28] interesting pod/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Sep 08 11:47:43.940058 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:43.939884 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" podUID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Sep 08 11:47:44.943733 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:44.943700 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:44.944146 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:44.943971 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:44.944628 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:44.944613 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:47:49.842730 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:49.842644 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:47:52.320556 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:52.320514 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 11:47:52.320978 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:52.320587 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 11:47:53.389549 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:53.389513 2564 generic.go:356] "Generic (PLEG): container finished" podID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerID="8e0d6905b20f1f7decc23309293ac2b4a027b8eca0dda96653129498384de662" exitCode=0 Sep 08 11:47:53.389994 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:53.389582 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" event={"ID":"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff","Type":"ContainerDied","Data":"8e0d6905b20f1f7decc23309293ac2b4a027b8eca0dda96653129498384de662"} Sep 08 11:47:53.389994 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:53.389972 2564 scope.go:117] "RemoveContainer" containerID="8e0d6905b20f1f7decc23309293ac2b4a027b8eca0dda96653129498384de662" Sep 08 11:47:53.391139 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:53.391111 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:53.391205 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:53.391179 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:53.391242 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:53.391210 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:47:54.397972 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:54.397938 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" event={"ID":"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff","Type":"ContainerStarted","Data":"6b4f2f4c0acb85ffe53f63b66152119750473df2ae932457abe00e2a1d787b00"} Sep 08 11:47:54.827600 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:54.827510 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:47:54.827755 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:54.827713 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 08 11:47:54.827804 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:54.827759 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 08 11:47:55.828225 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:55.828185 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 08 11:47:55.828629 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:55.828243 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 08 11:47:56.828294 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:56.828251 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 08 11:47:56.828789 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:56.828326 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 08 11:47:57.828297 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:57.828260 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 08 11:47:57.828700 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:57.828320 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 08 11:47:58.827558 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:58.827524 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 08 11:47:58.827744 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:58.827579 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 08 11:47:59.828314 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:59.828274 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 08 11:47:59.828724 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:47:59.828332 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 08 11:48:00.831066 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:00.831035 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:48:00.831526 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:00.831308 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:48:00.832014 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:00.831998 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:48:03.804813 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:03.804778 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r"] Sep 08 11:48:03.805263 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:03.805048 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" podUID="158fca76-814c-4abc-95a8-dce964ef4264" containerName="maas-api" containerID="cri-o://ed50a11882d84498194c35a6a8f026baeaff6e64126dfb91a3ff921f45a36e42" gracePeriod=30 Sep 08 11:48:04.056031 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.055966 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:48:04.154071 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.154033 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158fca76-814c-4abc-95a8-dce964ef4264-metrics-certs\") pod \"158fca76-814c-4abc-95a8-dce964ef4264\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " Sep 08 11:48:04.154266 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.154092 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwhcc\" (UniqueName: \"kubernetes.io/projected/158fca76-814c-4abc-95a8-dce964ef4264-kube-api-access-cwhcc\") pod \"158fca76-814c-4abc-95a8-dce964ef4264\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " Sep 08 11:48:04.154266 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.154137 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/158fca76-814c-4abc-95a8-dce964ef4264-openshift-service-ca\") pod \"158fca76-814c-4abc-95a8-dce964ef4264\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " Sep 08 11:48:04.154266 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.154164 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/158fca76-814c-4abc-95a8-dce964ef4264-maas-api-tls\") pod \"158fca76-814c-4abc-95a8-dce964ef4264\" (UID: \"158fca76-814c-4abc-95a8-dce964ef4264\") " Sep 08 11:48:04.154545 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.154517 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158fca76-814c-4abc-95a8-dce964ef4264-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "158fca76-814c-4abc-95a8-dce964ef4264" (UID: "158fca76-814c-4abc-95a8-dce964ef4264"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:48:04.156297 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.156263 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/158fca76-814c-4abc-95a8-dce964ef4264-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "158fca76-814c-4abc-95a8-dce964ef4264" (UID: "158fca76-814c-4abc-95a8-dce964ef4264"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:48:04.156458 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.156359 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/158fca76-814c-4abc-95a8-dce964ef4264-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "158fca76-814c-4abc-95a8-dce964ef4264" (UID: "158fca76-814c-4abc-95a8-dce964ef4264"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:48:04.156755 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.156734 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158fca76-814c-4abc-95a8-dce964ef4264-kube-api-access-cwhcc" (OuterVolumeSpecName: "kube-api-access-cwhcc") pod "158fca76-814c-4abc-95a8-dce964ef4264" (UID: "158fca76-814c-4abc-95a8-dce964ef4264"). InnerVolumeSpecName "kube-api-access-cwhcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:48:04.255926 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.255896 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/158fca76-814c-4abc-95a8-dce964ef4264-openshift-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:48:04.255926 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.255924 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/158fca76-814c-4abc-95a8-dce964ef4264-maas-api-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:48:04.255926 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.255933 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158fca76-814c-4abc-95a8-dce964ef4264-metrics-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:48:04.255926 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.255941 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwhcc\" (UniqueName: \"kubernetes.io/projected/158fca76-814c-4abc-95a8-dce964ef4264-kube-api-access-cwhcc\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:48:04.461308 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.461271 2564 generic.go:356] "Generic (PLEG): container finished" podID="158fca76-814c-4abc-95a8-dce964ef4264" containerID="ed50a11882d84498194c35a6a8f026baeaff6e64126dfb91a3ff921f45a36e42" exitCode=0 Sep 08 11:48:04.461489 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.461326 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" Sep 08 11:48:04.461489 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.461357 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" event={"ID":"158fca76-814c-4abc-95a8-dce964ef4264","Type":"ContainerDied","Data":"ed50a11882d84498194c35a6a8f026baeaff6e64126dfb91a3ff921f45a36e42"} Sep 08 11:48:04.461489 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.461451 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r" event={"ID":"158fca76-814c-4abc-95a8-dce964ef4264","Type":"ContainerDied","Data":"2d8c22dc20b996a39f005713543c9a2b9ddc4f997ea4842e26e40510dc65152e"} Sep 08 11:48:04.461489 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.461475 2564 scope.go:117] "RemoveContainer" containerID="ed50a11882d84498194c35a6a8f026baeaff6e64126dfb91a3ff921f45a36e42" Sep 08 11:48:04.471535 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.471517 2564 scope.go:117] "RemoveContainer" containerID="ed50a11882d84498194c35a6a8f026baeaff6e64126dfb91a3ff921f45a36e42" Sep 08 11:48:04.471769 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:48:04.471750 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed50a11882d84498194c35a6a8f026baeaff6e64126dfb91a3ff921f45a36e42\": container with ID starting with ed50a11882d84498194c35a6a8f026baeaff6e64126dfb91a3ff921f45a36e42 not found: ID does not exist" containerID="ed50a11882d84498194c35a6a8f026baeaff6e64126dfb91a3ff921f45a36e42" Sep 08 11:48:04.471828 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.471779 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed50a11882d84498194c35a6a8f026baeaff6e64126dfb91a3ff921f45a36e42"} err="failed to get container status \"ed50a11882d84498194c35a6a8f026baeaff6e64126dfb91a3ff921f45a36e42\": rpc error: code = NotFound desc = could not find container \"ed50a11882d84498194c35a6a8f026baeaff6e64126dfb91a3ff921f45a36e42\": container with ID starting with ed50a11882d84498194c35a6a8f026baeaff6e64126dfb91a3ff921f45a36e42 not found: ID does not exist" Sep 08 11:48:04.485513 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.485488 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r"] Sep 08 11:48:04.489695 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:04.489669 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2fe44c90-85479c6565-vsr6r"] Sep 08 11:48:05.652759 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:05.652727 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="158fca76-814c-4abc-95a8-dce964ef4264" path="/var/lib/kubelet/pods/158fca76-814c-4abc-95a8-dce964ef4264/volumes" Sep 08 11:48:07.542814 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:07.542780 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:48:11.401379 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:11.401331 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm"] Sep 08 11:48:11.401852 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:11.401596 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" podUID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" containerName="istio-proxy" containerID="cri-o://b4de367c300d463874152cfdd0d493bb80743f191b0e05a402626d446240a592" gracePeriod=30 Sep 08 11:48:14.946334 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:14.946290 2564 patch_prober.go:28] interesting pod/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 11:48:14.946802 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:14.946361 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" podUID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 11:48:16.541614 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.541580 2564 generic.go:356] "Generic (PLEG): container finished" podID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" containerID="b4de367c300d463874152cfdd0d493bb80743f191b0e05a402626d446240a592" exitCode=0 Sep 08 11:48:16.542066 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.541659 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" event={"ID":"2cab4e58-85f1-4e71-bc6a-87a67461bdd1","Type":"ContainerDied","Data":"b4de367c300d463874152cfdd0d493bb80743f191b0e05a402626d446240a592"} Sep 08 11:48:16.542066 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.541744 2564 scope.go:117] "RemoveContainer" containerID="2bfcf014058d47d9279225331e4a89391e109a4d68fb1887fc913ae16bdc2715" Sep 08 11:48:16.652306 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.652284 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:48:16.775747 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.775660 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-envoy\") pod \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " Sep 08 11:48:16.775747 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.775708 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-workload-socket\") pod \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " Sep 08 11:48:16.775747 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.775738 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-data\") pod \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " Sep 08 11:48:16.776033 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.775760 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-825wc\" (UniqueName: \"kubernetes.io/projected/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-kube-api-access-825wc\") pod \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " Sep 08 11:48:16.776033 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.775812 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-podinfo\") pod \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " Sep 08 11:48:16.776033 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.775844 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istiod-ca-cert\") pod \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " Sep 08 11:48:16.776033 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.775882 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-workload-certs\") pod \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " Sep 08 11:48:16.776033 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.775931 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-token\") pod \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " Sep 08 11:48:16.776033 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.775982 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-credential-socket\") pod \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\" (UID: \"2cab4e58-85f1-4e71-bc6a-87a67461bdd1\") " Sep 08 11:48:16.776033 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.776008 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2cab4e58-85f1-4e71-bc6a-87a67461bdd1" (UID: "2cab4e58-85f1-4e71-bc6a-87a67461bdd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:48:16.776445 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.776036 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-data" (OuterVolumeSpecName: "istio-data") pod "2cab4e58-85f1-4e71-bc6a-87a67461bdd1" (UID: "2cab4e58-85f1-4e71-bc6a-87a67461bdd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:48:16.776445 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.776215 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2cab4e58-85f1-4e71-bc6a-87a67461bdd1" (UID: "2cab4e58-85f1-4e71-bc6a-87a67461bdd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:48:16.776445 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.776321 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-workload-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:48:16.776445 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.776339 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-workload-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:48:16.776445 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.776352 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-data\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:48:16.776445 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.776358 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2cab4e58-85f1-4e71-bc6a-87a67461bdd1" (UID: "2cab4e58-85f1-4e71-bc6a-87a67461bdd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:48:16.776749 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.776562 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2cab4e58-85f1-4e71-bc6a-87a67461bdd1" (UID: "2cab4e58-85f1-4e71-bc6a-87a67461bdd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:48:16.778261 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.778223 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-kube-api-access-825wc" (OuterVolumeSpecName: "kube-api-access-825wc") pod "2cab4e58-85f1-4e71-bc6a-87a67461bdd1" (UID: "2cab4e58-85f1-4e71-bc6a-87a67461bdd1"). InnerVolumeSpecName "kube-api-access-825wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:48:16.778436 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.778259 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2cab4e58-85f1-4e71-bc6a-87a67461bdd1" (UID: "2cab4e58-85f1-4e71-bc6a-87a67461bdd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:48:16.778509 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.778456 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2cab4e58-85f1-4e71-bc6a-87a67461bdd1" (UID: "2cab4e58-85f1-4e71-bc6a-87a67461bdd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 11:48:16.778509 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.778476 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-token" (OuterVolumeSpecName: "istio-token") pod "2cab4e58-85f1-4e71-bc6a-87a67461bdd1" (UID: "2cab4e58-85f1-4e71-bc6a-87a67461bdd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:48:16.877884 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.877852 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-825wc\" (UniqueName: \"kubernetes.io/projected/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-kube-api-access-825wc\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:48:16.877884 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.877883 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-podinfo\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:48:16.878059 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.877897 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istiod-ca-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:48:16.878059 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.877911 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:48:16.878059 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.877924 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-credential-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:48:16.878059 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:16.877938 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cab4e58-85f1-4e71-bc6a-87a67461bdd1-istio-envoy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:48:17.548558 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:17.548528 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" Sep 08 11:48:17.548958 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:17.548571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm" event={"ID":"2cab4e58-85f1-4e71-bc6a-87a67461bdd1","Type":"ContainerDied","Data":"76ee93029e48d0fdb89fed5aa02297d83bbbb99120ee69a06135a33c74177cf4"} Sep 08 11:48:17.548958 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:17.548641 2564 scope.go:117] "RemoveContainer" containerID="b4de367c300d463874152cfdd0d493bb80743f191b0e05a402626d446240a592" Sep 08 11:48:17.573470 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:17.573443 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm"] Sep 08 11:48:17.580596 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:17.580573 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2fe44c90-openshift-default-86f59b487b-hmwrm"] Sep 08 11:48:17.653497 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:17.653466 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" path="/var/lib/kubelet/pods/2cab4e58-85f1-4e71-bc6a-87a67461bdd1/volumes" Sep 08 11:48:18.144760 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:18.144725 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:48:26.396555 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.396522 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc"] Sep 08 11:48:26.397092 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.397076 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" containerName="istio-proxy" Sep 08 11:48:26.397159 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.397095 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" containerName="istio-proxy" Sep 08 11:48:26.397159 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.397127 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="158fca76-814c-4abc-95a8-dce964ef4264" containerName="maas-api" Sep 08 11:48:26.397159 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.397135 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="158fca76-814c-4abc-95a8-dce964ef4264" containerName="maas-api" Sep 08 11:48:26.397316 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.397233 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" containerName="istio-proxy" Sep 08 11:48:26.397316 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.397247 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="158fca76-814c-4abc-95a8-dce964ef4264" containerName="maas-api" Sep 08 11:48:26.397316 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.397259 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" containerName="istio-proxy" Sep 08 11:48:26.400671 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.400648 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.403450 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.403424 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-69606d-openshift-default-dockercfg-rn78f\"" Sep 08 11:48:26.411590 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.411563 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc"] Sep 08 11:48:26.471924 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.471887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-credential-socket\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.472084 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.471935 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-envoy\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.472084 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.471989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkpxm\" (UniqueName: \"kubernetes.io/projected/6b3c3e85-998c-4d66-b921-0c97dab30697-kube-api-access-mkpxm\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.472084 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.472033 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-data\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.472084 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.472073 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-podinfo\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.472252 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.472119 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-workload-certs\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.472252 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.472150 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b3c3e85-998c-4d66-b921-0c97dab30697-istiod-ca-cert\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.472252 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.472188 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-workload-socket\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.472252 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.472220 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-token\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.573295 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.573254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-workload-certs\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.573527 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.573302 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b3c3e85-998c-4d66-b921-0c97dab30697-istiod-ca-cert\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.573527 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.573333 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-workload-socket\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.573527 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.573359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-token\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.573527 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.573495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-credential-socket\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.573527 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.573519 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-envoy\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.573798 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.573546 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkpxm\" (UniqueName: \"kubernetes.io/projected/6b3c3e85-998c-4d66-b921-0c97dab30697-kube-api-access-mkpxm\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.573798 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.573575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-data\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.573798 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.573613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-podinfo\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.573798 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.573739 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-workload-certs\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.574007 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.573910 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-credential-socket\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.574007 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.573921 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-workload-socket\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.574124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.574048 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-data\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.574325 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.574305 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b3c3e85-998c-4d66-b921-0c97dab30697-istiod-ca-cert\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.576053 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.576032 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-envoy\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.576601 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.576583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-podinfo\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.583267 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.583236 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkpxm\" (UniqueName: \"kubernetes.io/projected/6b3c3e85-998c-4d66-b921-0c97dab30697-kube-api-access-mkpxm\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.583495 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.583477 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-token\") pod \"e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.712402 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.712299 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:26.993643 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.993614 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc"] Sep 08 11:48:26.995264 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:48:26.995235 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3c3e85_998c_4d66_b921_0c97dab30697.slice/crio-c8852a278aa8f89ea73e7fc80ca6bcc0a6b614b8b8f3f76cf06f4456d228c578 WatchSource:0}: Error finding container c8852a278aa8f89ea73e7fc80ca6bcc0a6b614b8b8f3f76cf06f4456d228c578: Status 404 returned error can't find the container with id c8852a278aa8f89ea73e7fc80ca6bcc0a6b614b8b8f3f76cf06f4456d228c578 Sep 08 11:48:26.997560 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.997530 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:48:26.997672 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.997587 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:48:26.997672 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:26.997618 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:48:27.612927 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:27.612838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" event={"ID":"6b3c3e85-998c-4d66-b921-0c97dab30697","Type":"ContainerStarted","Data":"53c3f61dd8e280dc52e5525371030498bc48a6d3c909a4a6439a49e9708521cc"} Sep 08 11:48:27.612927 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:27.612902 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" event={"ID":"6b3c3e85-998c-4d66-b921-0c97dab30697","Type":"ContainerStarted","Data":"c8852a278aa8f89ea73e7fc80ca6bcc0a6b614b8b8f3f76cf06f4456d228c578"} Sep 08 11:48:27.637247 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:27.637196 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" podStartSLOduration=1.6371770799999998 podStartE2EDuration="1.63717708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:48:27.634735079 +0000 UTC m=+1096.493922412" watchObservedRunningTime="2026-09-08 11:48:27.63717708 +0000 UTC m=+1096.496364397" Sep 08 11:48:27.712548 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:27.712517 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:28.718229 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:28.718192 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:29.623568 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:29.623537 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:29.624454 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:29.624434 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:32.409494 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.409459 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl"] Sep 08 11:48:32.409883 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.409868 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" containerName="istio-proxy" Sep 08 11:48:32.409930 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.409885 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cab4e58-85f1-4e71-bc6a-87a67461bdd1" containerName="istio-proxy" Sep 08 11:48:32.412772 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.412748 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.416545 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.416523 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bb9b64\"/\"default-dockercfg-bbswc\"" Sep 08 11:48:32.416690 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.416545 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bb9b64\"/\"kube-root-ca.crt\"" Sep 08 11:48:32.416690 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.416547 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bb9b64\"/\"auth-test-model-bb9b64-kserve-self-signed-certs\"" Sep 08 11:48:32.416820 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.416575 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bb9b64\"/\"openshift-service-ca.crt\"" Sep 08 11:48:32.424211 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.424186 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl"] Sep 08 11:48:32.532080 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.532035 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-home\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.532080 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.532090 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-tmp-dir\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.532277 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.532122 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-dshm\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.532277 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.532146 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85d414af-6370-4683-82e9-5ffe0d2177d8-tls-certs\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.532277 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.532181 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-model-cache\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.532277 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.532231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8sjm\" (UniqueName: \"kubernetes.io/projected/85d414af-6370-4683-82e9-5ffe0d2177d8-kube-api-access-l8sjm\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.633257 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.633221 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-home\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.633257 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.633261 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-tmp-dir\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.633536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.633283 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-dshm\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.633536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.633306 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85d414af-6370-4683-82e9-5ffe0d2177d8-tls-certs\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.633536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.633340 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-model-cache\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.633536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.633361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8sjm\" (UniqueName: \"kubernetes.io/projected/85d414af-6370-4683-82e9-5ffe0d2177d8-kube-api-access-l8sjm\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.633756 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.633702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-home\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.633756 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.633713 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-tmp-dir\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.633837 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.633773 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-model-cache\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.635598 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.635574 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-dshm\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.635731 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.635711 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85d414af-6370-4683-82e9-5ffe0d2177d8-tls-certs\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.642625 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.642607 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8sjm\" (UniqueName: \"kubernetes.io/projected/85d414af-6370-4683-82e9-5ffe0d2177d8-kube-api-access-l8sjm\") pod \"auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:32.724418 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:32.724329 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:33.000737 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:33.000704 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl"] Sep 08 11:48:33.001847 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:48:33.001820 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85d414af_6370_4683_82e9_5ffe0d2177d8.slice/crio-f27bda6fbc91ca84cf26791b104f15c63533e2ba76a5573d4d60d2ee5f2a0981 WatchSource:0}: Error finding container f27bda6fbc91ca84cf26791b104f15c63533e2ba76a5573d4d60d2ee5f2a0981: Status 404 returned error can't find the container with id f27bda6fbc91ca84cf26791b104f15c63533e2ba76a5573d4d60d2ee5f2a0981 Sep 08 11:48:33.652979 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:33.652948 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" event={"ID":"85d414af-6370-4683-82e9-5ffe0d2177d8","Type":"ContainerStarted","Data":"94c9df75e613997ec87a419a90943f60b576e7bb5e5c53576c1a047342001f1a"} Sep 08 11:48:33.653335 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:33.652988 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" event={"ID":"85d414af-6370-4683-82e9-5ffe0d2177d8","Type":"ContainerStarted","Data":"f27bda6fbc91ca84cf26791b104f15c63533e2ba76a5573d4d60d2ee5f2a0981"} Sep 08 11:48:33.669974 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:33.669925 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" podStartSLOduration=1.669909229 podStartE2EDuration="1.669909229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:48:33.667807226 +0000 UTC m=+1102.526994545" watchObservedRunningTime="2026-09-08 11:48:33.669909229 +0000 UTC m=+1102.529096547" Sep 08 11:48:36.742205 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:36.742167 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:48:42.724963 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:42.724927 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:42.725424 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:42.725190 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:42.737992 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:42.737966 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:43.734855 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:43.734824 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:48:45.665579 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:45.665536 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 11:48:45.665969 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:45.665641 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" podUID="6b3c3e85-998c-4d66-b921-0c97dab30697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 11:48:46.741695 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:46.741600 2564 generic.go:356] "Generic (PLEG): container finished" podID="6b3c3e85-998c-4d66-b921-0c97dab30697" containerID="53c3f61dd8e280dc52e5525371030498bc48a6d3c909a4a6439a49e9708521cc" exitCode=0 Sep 08 11:48:46.741695 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:46.741673 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" event={"ID":"6b3c3e85-998c-4d66-b921-0c97dab30697","Type":"ContainerDied","Data":"53c3f61dd8e280dc52e5525371030498bc48a6d3c909a4a6439a49e9708521cc"} Sep 08 11:48:46.742121 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:46.742082 2564 scope.go:117] "RemoveContainer" containerID="53c3f61dd8e280dc52e5525371030498bc48a6d3c909a4a6439a49e9708521cc" Sep 08 11:48:46.743170 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:46.743139 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:48:46.743243 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:46.743209 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:48:46.743243 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:46.743237 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:48:47.749975 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:47.749937 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" event={"ID":"6b3c3e85-998c-4d66-b921-0c97dab30697","Type":"ContainerStarted","Data":"5f76f755fe6f583585bd666d25386ed225fd53464abb91f0555f6952c35539b8"} Sep 08 11:48:48.712782 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:48.712748 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:48.713015 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:48.712991 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 08 11:48:48.713063 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:48.713045 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" podUID="6b3c3e85-998c-4d66-b921-0c97dab30697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 08 11:48:49.713659 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:49.713623 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 08 11:48:49.714052 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:49.713679 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" podUID="6b3c3e85-998c-4d66-b921-0c97dab30697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 08 11:48:50.712743 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:50.712702 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 08 11:48:50.712924 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:50.712761 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" podUID="6b3c3e85-998c-4d66-b921-0c97dab30697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 08 11:48:51.713585 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:51.713545 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 08 11:48:51.713979 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:51.713618 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" podUID="6b3c3e85-998c-4d66-b921-0c97dab30697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 08 11:48:52.713354 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:52.713318 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 08 11:48:52.713561 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:52.713406 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" podUID="6b3c3e85-998c-4d66-b921-0c97dab30697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 08 11:48:53.716467 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:53.716435 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:53.716941 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:53.716819 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:53.717468 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:53.717448 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:48:54.984676 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:54.984638 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:48:59.519152 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.519109 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk"] Sep 08 11:48:59.523742 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.523715 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:48:59.526663 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.526636 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 08 11:48:59.527740 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.527716 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-489cdd-serving-cert\"" Sep 08 11:48:59.527862 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.527747 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b5t6p\"" Sep 08 11:48:59.527862 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.527757 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 11:48:59.527862 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.527718 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 08 11:48:59.532256 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.532234 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk"] Sep 08 11:48:59.617757 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.617722 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5khvx\" (UniqueName: \"kubernetes.io/projected/5a0ea850-5f36-482b-9719-58c6f16a395e-kube-api-access-5khvx\") pod \"maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:48:59.617914 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.617766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0ea850-5f36-482b-9719-58c6f16a395e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:48:59.617914 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.617852 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0ea850-5f36-482b-9719-58c6f16a395e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:48:59.617989 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.617921 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0ea850-5f36-482b-9719-58c6f16a395e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:48:59.719446 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.719409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5khvx\" (UniqueName: \"kubernetes.io/projected/5a0ea850-5f36-482b-9719-58c6f16a395e-kube-api-access-5khvx\") pod \"maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:48:59.719623 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.719463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0ea850-5f36-482b-9719-58c6f16a395e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:48:59.719623 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.719529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0ea850-5f36-482b-9719-58c6f16a395e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:48:59.719623 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.719603 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0ea850-5f36-482b-9719-58c6f16a395e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:48:59.719792 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:48:59.719633 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-serving-cert: secret "maas-api-e2e-shared-b-w2-489cdd-serving-cert" not found Sep 08 11:48:59.719792 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:48:59.719705 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a0ea850-5f36-482b-9719-58c6f16a395e-maas-api-tls podName:5a0ea850-5f36-482b-9719-58c6f16a395e nodeName:}" failed. No retries permitted until 2026-09-08 11:49:00.2196893 +0000 UTC m=+1129.078876595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5a0ea850-5f36-482b-9719-58c6f16a395e-maas-api-tls") pod "maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" (UID: "5a0ea850-5f36-482b-9719-58c6f16a395e") : secret "maas-api-e2e-shared-b-w2-489cdd-serving-cert" not found Sep 08 11:48:59.720261 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.720240 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0ea850-5f36-482b-9719-58c6f16a395e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:48:59.722186 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.722158 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0ea850-5f36-482b-9719-58c6f16a395e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:48:59.729839 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:48:59.729817 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5khvx\" (UniqueName: \"kubernetes.io/projected/5a0ea850-5f36-482b-9719-58c6f16a395e-kube-api-access-5khvx\") pod \"maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:49:00.224522 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:00.224487 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0ea850-5f36-482b-9719-58c6f16a395e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:49:00.226884 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:00.226860 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0ea850-5f36-482b-9719-58c6f16a395e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:49:00.436328 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:00.436294 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:49:00.720066 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:00.720042 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk"] Sep 08 11:49:00.721171 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:49:00.721144 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0ea850_5f36_482b_9719_58c6f16a395e.slice/crio-f669aadf910ed4d3cc6471692fdba083177cc915bfe35ca844aad556ae40549d WatchSource:0}: Error finding container f669aadf910ed4d3cc6471692fdba083177cc915bfe35ca844aad556ae40549d: Status 404 returned error can't find the container with id f669aadf910ed4d3cc6471692fdba083177cc915bfe35ca844aad556ae40549d Sep 08 11:49:00.832998 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:00.832903 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" event={"ID":"5a0ea850-5f36-482b-9719-58c6f16a395e","Type":"ContainerStarted","Data":"f669aadf910ed4d3cc6471692fdba083177cc915bfe35ca844aad556ae40549d"} Sep 08 11:49:01.840609 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:01.840572 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" event={"ID":"5a0ea850-5f36-482b-9719-58c6f16a395e","Type":"ContainerStarted","Data":"70a7bed115cb1c5a5f3da06d3056b5c61c5dbf690ec8495663b3d3cf0dca8d0c"} Sep 08 11:49:01.840995 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:01.840712 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:49:01.863258 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:01.863208 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" podStartSLOduration=2.446567227 podStartE2EDuration="2.863195023s" totalImagesPullingTime="416.627796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:49:01.860694395 +0000 UTC m=+1130.719881742" watchObservedRunningTime="2026-09-08 11:49:01.863195023 +0000 UTC m=+1130.722382342" Sep 08 11:49:07.853932 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:07.853897 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:49:09.956941 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:09.956909 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:49:11.715787 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:11.715753 2564 scope.go:117] "RemoveContainer" containerID="6e190ca4dcd385947bd03f683cefcac3f7213582b1e59186a6b994a1a42f6d01" Sep 08 11:49:24.740663 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:24.740626 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:49:39.397461 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.397354 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs"] Sep 08 11:49:39.400921 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.400899 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.403860 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.403839 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-63d6ac92-openshift-default-dockercfg-v5g2j\"" Sep 08 11:49:39.416154 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.416130 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs"] Sep 08 11:49:39.504939 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.504900 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-workload-certs\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.505103 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.504961 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-workload-socket\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.505103 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.504985 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5499e973-b228-4457-9073-b338ca8fa427-istio-token\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.505103 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.505004 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-credential-socket\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.505103 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.505037 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-istio-data\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.505103 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.505088 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-istio-envoy\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.505277 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.505117 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq8dn\" (UniqueName: \"kubernetes.io/projected/5499e973-b228-4457-9073-b338ca8fa427-kube-api-access-vq8dn\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.505277 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.505180 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5499e973-b228-4457-9073-b338ca8fa427-istiod-ca-cert\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.505277 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.505209 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5499e973-b228-4457-9073-b338ca8fa427-istio-podinfo\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.544500 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.544468 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:49:39.606323 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.606290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5499e973-b228-4457-9073-b338ca8fa427-istio-podinfo\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.606543 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.606342 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-workload-certs\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.606543 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.606420 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-workload-socket\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.606543 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.606444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5499e973-b228-4457-9073-b338ca8fa427-istio-token\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.606713 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.606574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-credential-socket\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.606713 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.606643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-istio-data\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.606713 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.606691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-istio-envoy\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.606933 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.606736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq8dn\" (UniqueName: \"kubernetes.io/projected/5499e973-b228-4457-9073-b338ca8fa427-kube-api-access-vq8dn\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.606933 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.606802 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-workload-certs\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.606933 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.606831 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5499e973-b228-4457-9073-b338ca8fa427-istiod-ca-cert\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.606933 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.606852 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-workload-socket\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.607153 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.607076 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-credential-socket\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.607213 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.607170 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-istio-data\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.607565 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.607542 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5499e973-b228-4457-9073-b338ca8fa427-istiod-ca-cert\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.609248 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.609221 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-istio-envoy\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.609464 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.609448 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5499e973-b228-4457-9073-b338ca8fa427-istio-podinfo\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.619494 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.619471 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5499e973-b228-4457-9073-b338ca8fa427-istio-token\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.619619 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.619595 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq8dn\" (UniqueName: \"kubernetes.io/projected/5499e973-b228-4457-9073-b338ca8fa427-kube-api-access-vq8dn\") pod \"e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.711185 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.711118 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:39.988190 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.988161 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs"] Sep 08 11:49:39.989086 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:49:39.989063 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5499e973_b228_4457_9073_b338ca8fa427.slice/crio-fabb82e88ea9f431a05507bedbae37a116203d7fa95f805063fcddedcb0a32b1 WatchSource:0}: Error finding container fabb82e88ea9f431a05507bedbae37a116203d7fa95f805063fcddedcb0a32b1: Status 404 returned error can't find the container with id fabb82e88ea9f431a05507bedbae37a116203d7fa95f805063fcddedcb0a32b1 Sep 08 11:49:39.991608 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.991560 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:49:39.991730 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.991662 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:49:39.991730 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:39.991713 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:49:40.068417 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:40.068390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" event={"ID":"5499e973-b228-4457-9073-b338ca8fa427","Type":"ContainerStarted","Data":"fabb82e88ea9f431a05507bedbae37a116203d7fa95f805063fcddedcb0a32b1"} Sep 08 11:49:41.076515 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:41.076472 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" event={"ID":"5499e973-b228-4457-9073-b338ca8fa427","Type":"ContainerStarted","Data":"4c0b58c8c1598e4825d1963101ed22761ef6f27a06eae3449b4a03f01c453930"} Sep 08 11:49:41.104925 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:41.104872 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" podStartSLOduration=2.10485902 podStartE2EDuration="2.10485902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:49:41.102508877 +0000 UTC m=+1169.961696196" watchObservedRunningTime="2026-09-08 11:49:41.10485902 +0000 UTC m=+1169.964046329" Sep 08 11:49:41.711878 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:41.711846 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:41.717234 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:41.717209 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:42.082853 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:42.082771 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:42.083824 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:42.083805 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:49:58.124549 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:58.124502 2564 patch_prober.go:28] interesting pod/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 11:49:58.124965 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:49:58.124597 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" podUID="5499e973-b228-4457-9073-b338ca8fa427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 11:50:00.205932 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:00.205895 2564 generic.go:356] "Generic (PLEG): container finished" podID="5499e973-b228-4457-9073-b338ca8fa427" containerID="4c0b58c8c1598e4825d1963101ed22761ef6f27a06eae3449b4a03f01c453930" exitCode=0 Sep 08 11:50:00.206408 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:00.205965 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" event={"ID":"5499e973-b228-4457-9073-b338ca8fa427","Type":"ContainerDied","Data":"4c0b58c8c1598e4825d1963101ed22761ef6f27a06eae3449b4a03f01c453930"} Sep 08 11:50:00.206473 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:00.206429 2564 scope.go:117] "RemoveContainer" containerID="4c0b58c8c1598e4825d1963101ed22761ef6f27a06eae3449b4a03f01c453930" Sep 08 11:50:00.207546 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:00.207517 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:50:00.207611 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:00.207584 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:50:00.207652 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:00.207617 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:50:01.214002 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:01.213965 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" event={"ID":"5499e973-b228-4457-9073-b338ca8fa427","Type":"ContainerStarted","Data":"3746bf53ffbc4373240b55243bf87753a2985fc366a16081b69a0b85ebe4736a"} Sep 08 11:50:01.711317 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:01.711270 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:50:01.711543 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:01.711512 2564 patch_prober.go:28] interesting pod/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 08 11:50:01.711609 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:01.711585 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" podUID="5499e973-b228-4457-9073-b338ca8fa427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 08 11:50:02.648336 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:02.648305 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:50:02.712101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:02.712061 2564 patch_prober.go:28] interesting pod/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 08 11:50:02.712280 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:02.712121 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" podUID="5499e973-b228-4457-9073-b338ca8fa427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 08 11:50:03.711977 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:03.711936 2564 patch_prober.go:28] interesting pod/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 08 11:50:03.712412 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:03.712012 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" podUID="5499e973-b228-4457-9073-b338ca8fa427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 08 11:50:04.711821 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:04.711777 2564 patch_prober.go:28] interesting pod/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 08 11:50:04.712023 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:04.711883 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" podUID="5499e973-b228-4457-9073-b338ca8fa427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 08 11:50:05.711809 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:05.711765 2564 patch_prober.go:28] interesting pod/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 08 11:50:05.712011 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:05.711833 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" podUID="5499e973-b228-4457-9073-b338ca8fa427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 08 11:50:06.773225 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:06.773185 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:50:06.773721 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:06.773612 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:50:06.774417 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:06.774400 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:50:11.670132 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:11.670100 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 11:50:11.670628 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:11.670301 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 11:50:19.742781 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:19.742746 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:50:31.021066 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.021028 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9"] Sep 08 11:50:31.024893 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.024868 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.027667 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.027644 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-63d6ac92\"/\"openshift-service-ca.crt\"" Sep 08 11:50:31.028850 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.028830 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-63d6ac92\"/\"e2e-lifecycle-model-63d6ac92-kserve-self-signed-certs\"" Sep 08 11:50:31.028917 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.028837 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-63d6ac92\"/\"kube-root-ca.crt\"" Sep 08 11:50:31.028917 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.028882 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-63d6ac92\"/\"default-dockercfg-zfx4t\"" Sep 08 11:50:31.036098 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.036076 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9"] Sep 08 11:50:31.217283 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.217243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-tmp-dir\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.217482 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.217313 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-home\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.217482 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.217347 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss9fm\" (UniqueName: \"kubernetes.io/projected/49346c95-9a14-45c5-b36f-ab34203e0d1a-kube-api-access-ss9fm\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.217482 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.217416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-model-cache\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.217482 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.217433 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-dshm\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.217482 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.217457 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49346c95-9a14-45c5-b36f-ab34203e0d1a-tls-certs\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.318965 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.318876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-home\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.318965 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.318936 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss9fm\" (UniqueName: \"kubernetes.io/projected/49346c95-9a14-45c5-b36f-ab34203e0d1a-kube-api-access-ss9fm\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.319224 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.318992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-model-cache\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.319224 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.319017 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-dshm\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.319224 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.319057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49346c95-9a14-45c5-b36f-ab34203e0d1a-tls-certs\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.319224 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.319112 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-tmp-dir\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.319484 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.319339 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-home\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.319545 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.319495 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-tmp-dir\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.319603 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.319591 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-model-cache\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.322144 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.322115 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-dshm\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.322583 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.322552 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49346c95-9a14-45c5-b36f-ab34203e0d1a-tls-certs\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.329265 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.329229 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss9fm\" (UniqueName: \"kubernetes.io/projected/49346c95-9a14-45c5-b36f-ab34203e0d1a-kube-api-access-ss9fm\") pod \"e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.335785 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.335559 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:31.613414 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:31.613391 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9"] Sep 08 11:50:31.615526 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:50:31.615495 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49346c95_9a14_45c5_b36f_ab34203e0d1a.slice/crio-46eb84dbf4164185a9228d6fcaf4439e27f70ef2b41ec54b56241adc28dbbc24 WatchSource:0}: Error finding container 46eb84dbf4164185a9228d6fcaf4439e27f70ef2b41ec54b56241adc28dbbc24: Status 404 returned error can't find the container with id 46eb84dbf4164185a9228d6fcaf4439e27f70ef2b41ec54b56241adc28dbbc24 Sep 08 11:50:32.403324 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:32.403286 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" event={"ID":"49346c95-9a14-45c5-b36f-ab34203e0d1a","Type":"ContainerStarted","Data":"75741ddb04670c5d35db5dd6f7638a6d831ac0ef26425f36d8653ba8b63d2145"} Sep 08 11:50:32.403324 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:32.403328 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" event={"ID":"49346c95-9a14-45c5-b36f-ab34203e0d1a","Type":"ContainerStarted","Data":"46eb84dbf4164185a9228d6fcaf4439e27f70ef2b41ec54b56241adc28dbbc24"} Sep 08 11:50:32.422996 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:32.422939 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" podStartSLOduration=1.422924648 podStartE2EDuration="1.422924648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:50:32.421062618 +0000 UTC m=+1221.280249965" watchObservedRunningTime="2026-09-08 11:50:32.422924648 +0000 UTC m=+1221.282112010" Sep 08 11:50:37.143642 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:37.143609 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:50:41.336424 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:41.336385 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:41.336424 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:41.336433 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:41.349326 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:41.349302 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:41.470734 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:41.470704 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:50:51.751720 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:51.751683 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:50:55.229282 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:50:55.229241 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl"] Sep 08 11:51:08.952569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:08.952484 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:51:10.249379 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.249316 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" podUID="85d414af-6370-4683-82e9-5ffe0d2177d8" containerName="main" containerID="cri-o://94c9df75e613997ec87a419a90943f60b576e7bb5e5c53576c1a047342001f1a" gracePeriod=45 Sep 08 11:51:10.523634 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.523602 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:51:10.626707 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.626663 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85d414af-6370-4683-82e9-5ffe0d2177d8-tls-certs\") pod \"85d414af-6370-4683-82e9-5ffe0d2177d8\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " Sep 08 11:51:10.626898 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.626773 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-model-cache\") pod \"85d414af-6370-4683-82e9-5ffe0d2177d8\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " Sep 08 11:51:10.626898 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.626802 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-dshm\") pod \"85d414af-6370-4683-82e9-5ffe0d2177d8\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " Sep 08 11:51:10.626898 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.626840 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-home\") pod \"85d414af-6370-4683-82e9-5ffe0d2177d8\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " Sep 08 11:51:10.626898 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.626867 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8sjm\" (UniqueName: \"kubernetes.io/projected/85d414af-6370-4683-82e9-5ffe0d2177d8-kube-api-access-l8sjm\") pod \"85d414af-6370-4683-82e9-5ffe0d2177d8\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " Sep 08 11:51:10.627138 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.626902 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-tmp-dir\") pod \"85d414af-6370-4683-82e9-5ffe0d2177d8\" (UID: \"85d414af-6370-4683-82e9-5ffe0d2177d8\") " Sep 08 11:51:10.627138 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.627098 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "85d414af-6370-4683-82e9-5ffe0d2177d8" (UID: "85d414af-6370-4683-82e9-5ffe0d2177d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:51:10.627241 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.627126 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-home" (OuterVolumeSpecName: "home") pod "85d414af-6370-4683-82e9-5ffe0d2177d8" (UID: "85d414af-6370-4683-82e9-5ffe0d2177d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:51:10.627317 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.627287 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85d414af-6370-4683-82e9-5ffe0d2177d8" (UID: "85d414af-6370-4683-82e9-5ffe0d2177d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:51:10.627317 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.627300 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:51:10.627462 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.627320 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:51:10.629010 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.628986 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85d414af-6370-4683-82e9-5ffe0d2177d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85d414af-6370-4683-82e9-5ffe0d2177d8" (UID: "85d414af-6370-4683-82e9-5ffe0d2177d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:51:10.629148 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.629043 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-dshm" (OuterVolumeSpecName: "dshm") pod "85d414af-6370-4683-82e9-5ffe0d2177d8" (UID: "85d414af-6370-4683-82e9-5ffe0d2177d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:51:10.629210 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.629146 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d414af-6370-4683-82e9-5ffe0d2177d8-kube-api-access-l8sjm" (OuterVolumeSpecName: "kube-api-access-l8sjm") pod "85d414af-6370-4683-82e9-5ffe0d2177d8" (UID: "85d414af-6370-4683-82e9-5ffe0d2177d8"). InnerVolumeSpecName "kube-api-access-l8sjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:51:10.642397 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.642340 2564 generic.go:356] "Generic (PLEG): container finished" podID="85d414af-6370-4683-82e9-5ffe0d2177d8" containerID="94c9df75e613997ec87a419a90943f60b576e7bb5e5c53576c1a047342001f1a" exitCode=0 Sep 08 11:51:10.642509 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.642398 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" Sep 08 11:51:10.642509 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.642427 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" event={"ID":"85d414af-6370-4683-82e9-5ffe0d2177d8","Type":"ContainerDied","Data":"94c9df75e613997ec87a419a90943f60b576e7bb5e5c53576c1a047342001f1a"} Sep 08 11:51:10.642509 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.642474 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl" event={"ID":"85d414af-6370-4683-82e9-5ffe0d2177d8","Type":"ContainerDied","Data":"f27bda6fbc91ca84cf26791b104f15c63533e2ba76a5573d4d60d2ee5f2a0981"} Sep 08 11:51:10.642509 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.642491 2564 scope.go:117] "RemoveContainer" containerID="94c9df75e613997ec87a419a90943f60b576e7bb5e5c53576c1a047342001f1a" Sep 08 11:51:10.652715 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.652701 2564 scope.go:117] "RemoveContainer" containerID="94c9df75e613997ec87a419a90943f60b576e7bb5e5c53576c1a047342001f1a" Sep 08 11:51:10.652981 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:51:10.652958 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c9df75e613997ec87a419a90943f60b576e7bb5e5c53576c1a047342001f1a\": container with ID starting with 94c9df75e613997ec87a419a90943f60b576e7bb5e5c53576c1a047342001f1a not found: ID does not exist" containerID="94c9df75e613997ec87a419a90943f60b576e7bb5e5c53576c1a047342001f1a" Sep 08 11:51:10.653046 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.652994 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c9df75e613997ec87a419a90943f60b576e7bb5e5c53576c1a047342001f1a"} err="failed to get container status \"94c9df75e613997ec87a419a90943f60b576e7bb5e5c53576c1a047342001f1a\": rpc error: code = NotFound desc = could not find container \"94c9df75e613997ec87a419a90943f60b576e7bb5e5c53576c1a047342001f1a\": container with ID starting with 94c9df75e613997ec87a419a90943f60b576e7bb5e5c53576c1a047342001f1a not found: ID does not exist" Sep 08 11:51:10.669982 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.669959 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl"] Sep 08 11:51:10.675568 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.675547 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bb9b64/auth-test-model-bb9b64-kserve-84fcbb9975-2kfwl"] Sep 08 11:51:10.728015 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.727993 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8sjm\" (UniqueName: \"kubernetes.io/projected/85d414af-6370-4683-82e9-5ffe0d2177d8-kube-api-access-l8sjm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:51:10.728015 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.728013 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:51:10.728144 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.728024 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85d414af-6370-4683-82e9-5ffe0d2177d8-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:51:10.728144 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:10.728031 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85d414af-6370-4683-82e9-5ffe0d2177d8-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:51:11.654343 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:11.654313 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85d414af-6370-4683-82e9-5ffe0d2177d8" path="/var/lib/kubelet/pods/85d414af-6370-4683-82e9-5ffe0d2177d8/volumes" Sep 08 11:51:11.750243 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:11.750211 2564 scope.go:117] "RemoveContainer" containerID="742e0abdb3b26a6d90effc8dd60c1454c8de80ba5061c570b7dc8e20b7cf6825" Sep 08 11:51:26.843511 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:26.843475 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:51:31.874448 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:31.874416 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9"] Sep 08 11:51:41.939661 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:41.939622 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:51:46.893686 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:46.893641 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" podUID="49346c95-9a14-45c5-b36f-ab34203e0d1a" containerName="main" containerID="cri-o://75741ddb04670c5d35db5dd6f7638a6d831ac0ef26425f36d8653ba8b63d2145" gracePeriod=45 Sep 08 11:51:47.143589 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.143564 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:51:47.173336 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.173283 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-tmp-dir\") pod \"49346c95-9a14-45c5-b36f-ab34203e0d1a\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " Sep 08 11:51:47.173336 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.173332 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss9fm\" (UniqueName: \"kubernetes.io/projected/49346c95-9a14-45c5-b36f-ab34203e0d1a-kube-api-access-ss9fm\") pod \"49346c95-9a14-45c5-b36f-ab34203e0d1a\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " Sep 08 11:51:47.173503 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.173421 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-model-cache\") pod \"49346c95-9a14-45c5-b36f-ab34203e0d1a\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " Sep 08 11:51:47.173503 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.173458 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-home\") pod \"49346c95-9a14-45c5-b36f-ab34203e0d1a\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " Sep 08 11:51:47.173503 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.173490 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-dshm\") pod \"49346c95-9a14-45c5-b36f-ab34203e0d1a\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " Sep 08 11:51:47.173609 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.173512 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49346c95-9a14-45c5-b36f-ab34203e0d1a-tls-certs\") pod \"49346c95-9a14-45c5-b36f-ab34203e0d1a\" (UID: \"49346c95-9a14-45c5-b36f-ab34203e0d1a\") " Sep 08 11:51:47.173644 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.173604 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "49346c95-9a14-45c5-b36f-ab34203e0d1a" (UID: "49346c95-9a14-45c5-b36f-ab34203e0d1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:51:47.173861 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.173841 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:51:47.174077 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.174044 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-model-cache" (OuterVolumeSpecName: "model-cache") pod "49346c95-9a14-45c5-b36f-ab34203e0d1a" (UID: "49346c95-9a14-45c5-b36f-ab34203e0d1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:51:47.174510 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.174485 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-home" (OuterVolumeSpecName: "home") pod "49346c95-9a14-45c5-b36f-ab34203e0d1a" (UID: "49346c95-9a14-45c5-b36f-ab34203e0d1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:51:47.177052 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.176995 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49346c95-9a14-45c5-b36f-ab34203e0d1a-kube-api-access-ss9fm" (OuterVolumeSpecName: "kube-api-access-ss9fm") pod "49346c95-9a14-45c5-b36f-ab34203e0d1a" (UID: "49346c95-9a14-45c5-b36f-ab34203e0d1a"). InnerVolumeSpecName "kube-api-access-ss9fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:51:47.177052 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.177008 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49346c95-9a14-45c5-b36f-ab34203e0d1a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49346c95-9a14-45c5-b36f-ab34203e0d1a" (UID: "49346c95-9a14-45c5-b36f-ab34203e0d1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:51:47.177207 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.177058 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-dshm" (OuterVolumeSpecName: "dshm") pod "49346c95-9a14-45c5-b36f-ab34203e0d1a" (UID: "49346c95-9a14-45c5-b36f-ab34203e0d1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:51:47.274974 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.274945 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:51:47.274974 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.274971 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:51:47.274974 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.274988 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49346c95-9a14-45c5-b36f-ab34203e0d1a-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:51:47.275224 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.275003 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss9fm\" (UniqueName: \"kubernetes.io/projected/49346c95-9a14-45c5-b36f-ab34203e0d1a-kube-api-access-ss9fm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:51:47.275224 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.275016 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49346c95-9a14-45c5-b36f-ab34203e0d1a-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:51:47.868665 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.868630 2564 generic.go:356] "Generic (PLEG): container finished" podID="49346c95-9a14-45c5-b36f-ab34203e0d1a" containerID="75741ddb04670c5d35db5dd6f7638a6d831ac0ef26425f36d8653ba8b63d2145" exitCode=0 Sep 08 11:51:47.868858 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.868671 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" Sep 08 11:51:47.868858 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.868722 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" event={"ID":"49346c95-9a14-45c5-b36f-ab34203e0d1a","Type":"ContainerDied","Data":"75741ddb04670c5d35db5dd6f7638a6d831ac0ef26425f36d8653ba8b63d2145"} Sep 08 11:51:47.868858 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.868771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9" event={"ID":"49346c95-9a14-45c5-b36f-ab34203e0d1a","Type":"ContainerDied","Data":"46eb84dbf4164185a9228d6fcaf4439e27f70ef2b41ec54b56241adc28dbbc24"} Sep 08 11:51:47.868858 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.868796 2564 scope.go:117] "RemoveContainer" containerID="75741ddb04670c5d35db5dd6f7638a6d831ac0ef26425f36d8653ba8b63d2145" Sep 08 11:51:47.878005 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.877987 2564 scope.go:117] "RemoveContainer" containerID="75741ddb04670c5d35db5dd6f7638a6d831ac0ef26425f36d8653ba8b63d2145" Sep 08 11:51:47.878284 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:51:47.878263 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75741ddb04670c5d35db5dd6f7638a6d831ac0ef26425f36d8653ba8b63d2145\": container with ID starting with 75741ddb04670c5d35db5dd6f7638a6d831ac0ef26425f36d8653ba8b63d2145 not found: ID does not exist" containerID="75741ddb04670c5d35db5dd6f7638a6d831ac0ef26425f36d8653ba8b63d2145" Sep 08 11:51:47.878393 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.878292 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75741ddb04670c5d35db5dd6f7638a6d831ac0ef26425f36d8653ba8b63d2145"} err="failed to get container status \"75741ddb04670c5d35db5dd6f7638a6d831ac0ef26425f36d8653ba8b63d2145\": rpc error: code = NotFound desc = could not find container \"75741ddb04670c5d35db5dd6f7638a6d831ac0ef26425f36d8653ba8b63d2145\": container with ID starting with 75741ddb04670c5d35db5dd6f7638a6d831ac0ef26425f36d8653ba8b63d2145 not found: ID does not exist" Sep 08 11:51:47.888684 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.888661 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9"] Sep 08 11:51:47.892904 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:47.892884 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-63d6ac92/e2e-lifecycle-model-63d6ac92-kserve-6444f989c9-l24m9"] Sep 08 11:51:49.653618 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:49.653582 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49346c95-9a14-45c5-b36f-ab34203e0d1a" path="/var/lib/kubelet/pods/49346c95-9a14-45c5-b36f-ab34203e0d1a/volumes" Sep 08 11:51:53.844017 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:51:53.843983 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:52:10.741118 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:10.741080 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:52:13.383450 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:13.383414 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7440-015fb0"] Sep 08 11:52:13.383833 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:13.383824 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85d414af-6370-4683-82e9-5ffe0d2177d8" containerName="main" Sep 08 11:52:13.383877 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:13.383836 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="85d414af-6370-4683-82e9-5ffe0d2177d8" containerName="main" Sep 08 11:52:13.383877 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:13.383845 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49346c95-9a14-45c5-b36f-ab34203e0d1a" containerName="main" Sep 08 11:52:13.383877 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:13.383851 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="49346c95-9a14-45c5-b36f-ab34203e0d1a" containerName="main" Sep 08 11:52:13.383982 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:13.383915 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="85d414af-6370-4683-82e9-5ffe0d2177d8" containerName="main" Sep 08 11:52:13.383982 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:13.383923 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="49346c95-9a14-45c5-b36f-ab34203e0d1a" containerName="main" Sep 08 11:52:13.387212 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:13.387195 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7440-015fb0" Sep 08 11:52:13.390284 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:13.390262 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zbjsb\"" Sep 08 11:52:13.397969 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:13.397948 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7440-015fb0"] Sep 08 11:52:13.533049 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:13.533014 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j955p\" (UniqueName: \"kubernetes.io/projected/1336d9f6-d926-4f4a-8474-dd86cfd85b6c-kube-api-access-j955p\") pod \"mt-curl-7440-015fb0\" (UID: \"1336d9f6-d926-4f4a-8474-dd86cfd85b6c\") " pod="opendatahub/mt-curl-7440-015fb0" Sep 08 11:52:13.634101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:13.634016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j955p\" (UniqueName: \"kubernetes.io/projected/1336d9f6-d926-4f4a-8474-dd86cfd85b6c-kube-api-access-j955p\") pod \"mt-curl-7440-015fb0\" (UID: \"1336d9f6-d926-4f4a-8474-dd86cfd85b6c\") " pod="opendatahub/mt-curl-7440-015fb0" Sep 08 11:52:13.644765 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:13.644737 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j955p\" (UniqueName: \"kubernetes.io/projected/1336d9f6-d926-4f4a-8474-dd86cfd85b6c-kube-api-access-j955p\") pod \"mt-curl-7440-015fb0\" (UID: \"1336d9f6-d926-4f4a-8474-dd86cfd85b6c\") " pod="opendatahub/mt-curl-7440-015fb0" Sep 08 11:52:13.696011 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:13.695986 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7440-015fb0" Sep 08 11:52:13.968067 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:13.968041 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7440-015fb0"] Sep 08 11:52:13.969584 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:52:13.969544 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1336d9f6_d926_4f4a_8474_dd86cfd85b6c.slice/crio-d032e2c47b6680ab70cb575d2e241456bc3b6bde6e5bf768cb9409eb012b7422 WatchSource:0}: Error finding container d032e2c47b6680ab70cb575d2e241456bc3b6bde6e5bf768cb9409eb012b7422: Status 404 returned error can't find the container with id d032e2c47b6680ab70cb575d2e241456bc3b6bde6e5bf768cb9409eb012b7422 Sep 08 11:52:14.026302 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:14.026274 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7440-015fb0" event={"ID":"1336d9f6-d926-4f4a-8474-dd86cfd85b6c","Type":"ContainerStarted","Data":"d032e2c47b6680ab70cb575d2e241456bc3b6bde6e5bf768cb9409eb012b7422"} Sep 08 11:52:15.033718 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:15.033635 2564 generic.go:356] "Generic (PLEG): container finished" podID="1336d9f6-d926-4f4a-8474-dd86cfd85b6c" containerID="196648f3d545599d456e5a420447ed53f9b37468c725a3bcf6b3cf7a036ae1ed" exitCode=0 Sep 08 11:52:15.034070 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:15.033718 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7440-015fb0" event={"ID":"1336d9f6-d926-4f4a-8474-dd86cfd85b6c","Type":"ContainerDied","Data":"196648f3d545599d456e5a420447ed53f9b37468c725a3bcf6b3cf7a036ae1ed"} Sep 08 11:52:16.172907 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.172886 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7440-015fb0" Sep 08 11:52:16.212178 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.212146 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7440-015fb0_1336d9f6-d926-4f4a-8474-dd86cfd85b6c/mt-curl-7440-015fb0/0.log" Sep 08 11:52:16.236483 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.236441 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7440-015fb0"] Sep 08 11:52:16.241017 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.240994 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7440-015fb0"] Sep 08 11:52:16.263320 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.263293 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j955p\" (UniqueName: \"kubernetes.io/projected/1336d9f6-d926-4f4a-8474-dd86cfd85b6c-kube-api-access-j955p\") pod \"1336d9f6-d926-4f4a-8474-dd86cfd85b6c\" (UID: \"1336d9f6-d926-4f4a-8474-dd86cfd85b6c\") " Sep 08 11:52:16.265308 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.265282 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1336d9f6-d926-4f4a-8474-dd86cfd85b6c-kube-api-access-j955p" (OuterVolumeSpecName: "kube-api-access-j955p") pod "1336d9f6-d926-4f4a-8474-dd86cfd85b6c" (UID: "1336d9f6-d926-4f4a-8474-dd86cfd85b6c"). InnerVolumeSpecName "kube-api-access-j955p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:52:16.357898 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.357823 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7440-beaeae"] Sep 08 11:52:16.358283 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.358269 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1336d9f6-d926-4f4a-8474-dd86cfd85b6c" containerName="mt-curl-7440-015fb0" Sep 08 11:52:16.358326 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.358286 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1336d9f6-d926-4f4a-8474-dd86cfd85b6c" containerName="mt-curl-7440-015fb0" Sep 08 11:52:16.358393 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.358382 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1336d9f6-d926-4f4a-8474-dd86cfd85b6c" containerName="mt-curl-7440-015fb0" Sep 08 11:52:16.361472 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.361453 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7440-beaeae" Sep 08 11:52:16.364497 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.364476 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j955p\" (UniqueName: \"kubernetes.io/projected/1336d9f6-d926-4f4a-8474-dd86cfd85b6c-kube-api-access-j955p\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:52:16.369604 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.369579 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7440-beaeae"] Sep 08 11:52:16.465546 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.465514 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c4vr\" (UniqueName: \"kubernetes.io/projected/7906f285-0da8-4040-a748-693b7d48e931-kube-api-access-9c4vr\") pod \"mt-curl-7440-beaeae\" (UID: \"7906f285-0da8-4040-a748-693b7d48e931\") " pod="opendatahub/mt-curl-7440-beaeae" Sep 08 11:52:16.566268 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.566237 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c4vr\" (UniqueName: \"kubernetes.io/projected/7906f285-0da8-4040-a748-693b7d48e931-kube-api-access-9c4vr\") pod \"mt-curl-7440-beaeae\" (UID: \"7906f285-0da8-4040-a748-693b7d48e931\") " pod="opendatahub/mt-curl-7440-beaeae" Sep 08 11:52:16.576961 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.576933 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c4vr\" (UniqueName: \"kubernetes.io/projected/7906f285-0da8-4040-a748-693b7d48e931-kube-api-access-9c4vr\") pod \"mt-curl-7440-beaeae\" (UID: \"7906f285-0da8-4040-a748-693b7d48e931\") " pod="opendatahub/mt-curl-7440-beaeae" Sep 08 11:52:16.672794 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.672765 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7440-beaeae" Sep 08 11:52:16.950520 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:16.950490 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7440-beaeae"] Sep 08 11:52:16.951570 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:52:16.951541 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7906f285_0da8_4040_a748_693b7d48e931.slice/crio-d08b41071e2378fcbe71231275837267a2c1561afa528b19ea4c73768d0b7f2e WatchSource:0}: Error finding container d08b41071e2378fcbe71231275837267a2c1561afa528b19ea4c73768d0b7f2e: Status 404 returned error can't find the container with id d08b41071e2378fcbe71231275837267a2c1561afa528b19ea4c73768d0b7f2e Sep 08 11:52:17.047948 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:17.047928 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7440-015fb0" Sep 08 11:52:17.048100 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:17.047959 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d032e2c47b6680ab70cb575d2e241456bc3b6bde6e5bf768cb9409eb012b7422" Sep 08 11:52:17.049288 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:17.049258 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7440-beaeae" event={"ID":"7906f285-0da8-4040-a748-693b7d48e931","Type":"ContainerStarted","Data":"d08b41071e2378fcbe71231275837267a2c1561afa528b19ea4c73768d0b7f2e"} Sep 08 11:52:17.059219 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:52:17.059196 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7440-015fb0\" is forbidden: User \"system:node:ip-10-0-133-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-187.ec2.internal' and this object" podUID="1336d9f6-d926-4f4a-8474-dd86cfd85b6c" pod="opendatahub/mt-curl-7440-015fb0" Sep 08 11:52:17.654284 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:17.654253 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1336d9f6-d926-4f4a-8474-dd86cfd85b6c" path="/var/lib/kubelet/pods/1336d9f6-d926-4f4a-8474-dd86cfd85b6c/volumes" Sep 08 11:52:18.057599 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:18.057519 2564 generic.go:356] "Generic (PLEG): container finished" podID="7906f285-0da8-4040-a748-693b7d48e931" containerID="80a8e4381e0d74eed1160496b906bb000dc3e0be59971387887d667d690e4f0e" exitCode=0 Sep 08 11:52:18.057599 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:18.057592 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7440-beaeae" event={"ID":"7906f285-0da8-4040-a748-693b7d48e931","Type":"ContainerDied","Data":"80a8e4381e0d74eed1160496b906bb000dc3e0be59971387887d667d690e4f0e"} Sep 08 11:52:19.190162 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:19.190140 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7440-beaeae" Sep 08 11:52:19.216499 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:19.216467 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7440-beaeae_7906f285-0da8-4040-a748-693b7d48e931/mt-curl-7440-beaeae/0.log" Sep 08 11:52:19.242017 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:19.241988 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7440-beaeae"] Sep 08 11:52:19.246422 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:19.246397 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7440-beaeae"] Sep 08 11:52:19.296587 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:19.296560 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c4vr\" (UniqueName: \"kubernetes.io/projected/7906f285-0da8-4040-a748-693b7d48e931-kube-api-access-9c4vr\") pod \"7906f285-0da8-4040-a748-693b7d48e931\" (UID: \"7906f285-0da8-4040-a748-693b7d48e931\") " Sep 08 11:52:19.298518 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:19.298495 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7906f285-0da8-4040-a748-693b7d48e931-kube-api-access-9c4vr" (OuterVolumeSpecName: "kube-api-access-9c4vr") pod "7906f285-0da8-4040-a748-693b7d48e931" (UID: "7906f285-0da8-4040-a748-693b7d48e931"). InnerVolumeSpecName "kube-api-access-9c4vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:52:19.397830 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:19.397803 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c4vr\" (UniqueName: \"kubernetes.io/projected/7906f285-0da8-4040-a748-693b7d48e931-kube-api-access-9c4vr\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:52:19.653109 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:19.653030 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7906f285-0da8-4040-a748-693b7d48e931" path="/var/lib/kubelet/pods/7906f285-0da8-4040-a748-693b7d48e931/volumes" Sep 08 11:52:20.069641 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:20.069555 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7440-beaeae" Sep 08 11:52:20.069641 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:20.069622 2564 scope.go:117] "RemoveContainer" containerID="80a8e4381e0d74eed1160496b906bb000dc3e0be59971387887d667d690e4f0e" Sep 08 11:52:24.844399 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:24.843432 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:52:38.949396 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:38.949297 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:52:41.277914 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:41.277878 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs"] Sep 08 11:52:41.278431 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:41.278098 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" podUID="5499e973-b228-4457-9073-b338ca8fa427" containerName="istio-proxy" containerID="cri-o://3746bf53ffbc4373240b55243bf87753a2985fc366a16081b69a0b85ebe4736a" gracePeriod=30 Sep 08 11:52:43.446598 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.446560 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f"] Sep 08 11:52:43.447206 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.447185 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7906f285-0da8-4040-a748-693b7d48e931" containerName="mt-curl-7440-beaeae" Sep 08 11:52:43.447206 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.447208 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7906f285-0da8-4040-a748-693b7d48e931" containerName="mt-curl-7440-beaeae" Sep 08 11:52:43.447405 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.447318 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7906f285-0da8-4040-a748-693b7d48e931" containerName="mt-curl-7440-beaeae" Sep 08 11:52:43.450219 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.450198 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.453079 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.453058 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-838a68\"/\"openshift-service-ca.crt\"" Sep 08 11:52:43.454393 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.454349 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-838a68\"/\"default-dockercfg-8k58w\"" Sep 08 11:52:43.454485 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.454349 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-838a68\"/\"rate-test-model-838a68-kserve-self-signed-certs\"" Sep 08 11:52:43.454485 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.454439 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-838a68\"/\"kube-root-ca.crt\"" Sep 08 11:52:43.460576 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.460554 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f"] Sep 08 11:52:43.532060 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.532030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-tmp-dir\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.532224 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.532087 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-dshm\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.532224 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.532163 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5db1c580-2330-4820-b4df-f0b86b55330f-tls-certs\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.532224 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.532197 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-home\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.532224 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.532219 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgzpm\" (UniqueName: \"kubernetes.io/projected/5db1c580-2330-4820-b4df-f0b86b55330f-kube-api-access-xgzpm\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.532437 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.532272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-model-cache\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.633312 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.633242 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-home\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.633652 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.633590 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgzpm\" (UniqueName: \"kubernetes.io/projected/5db1c580-2330-4820-b4df-f0b86b55330f-kube-api-access-xgzpm\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.633755 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.633709 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-home\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.634151 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.634106 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-model-cache\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.634338 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.634286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-model-cache\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.635120 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.634567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-tmp-dir\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.635120 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.634639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-dshm\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.635120 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.634694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5db1c580-2330-4820-b4df-f0b86b55330f-tls-certs\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.635908 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.635879 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-tmp-dir\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.638334 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.638290 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5db1c580-2330-4820-b4df-f0b86b55330f-tls-certs\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.638915 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.638869 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-dshm\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.643990 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.643965 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgzpm\" (UniqueName: \"kubernetes.io/projected/5db1c580-2330-4820-b4df-f0b86b55330f-kube-api-access-xgzpm\") pod \"rate-test-model-838a68-kserve-6d744c5fcf-4qt8f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:43.760223 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:43.760144 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:44.055305 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:44.055276 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f"] Sep 08 11:52:44.057135 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:52:44.057103 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5db1c580_2330_4820_b4df_f0b86b55330f.slice/crio-8226e8d83e3eb306605d0075f9db0dc05bb570cd846b2c3060f497a6c02d272b WatchSource:0}: Error finding container 8226e8d83e3eb306605d0075f9db0dc05bb570cd846b2c3060f497a6c02d272b: Status 404 returned error can't find the container with id 8226e8d83e3eb306605d0075f9db0dc05bb570cd846b2c3060f497a6c02d272b Sep 08 11:52:44.214094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:44.214060 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" event={"ID":"5db1c580-2330-4820-b4df-f0b86b55330f","Type":"ContainerStarted","Data":"e3467264c5196f459e4ea5b22eb1324442b5f0e658b4d00465e381ccfe2b98b2"} Sep 08 11:52:44.214261 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:44.214104 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" event={"ID":"5db1c580-2330-4820-b4df-f0b86b55330f","Type":"ContainerStarted","Data":"8226e8d83e3eb306605d0075f9db0dc05bb570cd846b2c3060f497a6c02d272b"} Sep 08 11:52:44.234989 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:44.234938 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" podStartSLOduration=1.234922291 podStartE2EDuration="1.234922291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:52:44.233402938 +0000 UTC m=+1353.092590281" watchObservedRunningTime="2026-09-08 11:52:44.234922291 +0000 UTC m=+1353.094109608" Sep 08 11:52:46.532568 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.532542 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:52:46.664377 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.664348 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq8dn\" (UniqueName: \"kubernetes.io/projected/5499e973-b228-4457-9073-b338ca8fa427-kube-api-access-vq8dn\") pod \"5499e973-b228-4457-9073-b338ca8fa427\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " Sep 08 11:52:46.664561 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.664408 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5499e973-b228-4457-9073-b338ca8fa427-istio-podinfo\") pod \"5499e973-b228-4457-9073-b338ca8fa427\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " Sep 08 11:52:46.664561 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.664477 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-workload-socket\") pod \"5499e973-b228-4457-9073-b338ca8fa427\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " Sep 08 11:52:46.664561 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.664517 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-istio-data\") pod \"5499e973-b228-4457-9073-b338ca8fa427\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " Sep 08 11:52:46.664561 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.664557 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-istio-envoy\") pod \"5499e973-b228-4457-9073-b338ca8fa427\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " Sep 08 11:52:46.664770 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.664583 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5499e973-b228-4457-9073-b338ca8fa427-istiod-ca-cert\") pod \"5499e973-b228-4457-9073-b338ca8fa427\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " Sep 08 11:52:46.664770 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.664626 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5499e973-b228-4457-9073-b338ca8fa427-istio-token\") pod \"5499e973-b228-4457-9073-b338ca8fa427\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " Sep 08 11:52:46.664770 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.664661 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-workload-certs\") pod \"5499e973-b228-4457-9073-b338ca8fa427\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " Sep 08 11:52:46.664770 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.664728 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-credential-socket\") pod \"5499e973-b228-4457-9073-b338ca8fa427\" (UID: \"5499e973-b228-4457-9073-b338ca8fa427\") " Sep 08 11:52:46.664965 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.664770 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5499e973-b228-4457-9073-b338ca8fa427" (UID: "5499e973-b228-4457-9073-b338ca8fa427"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:52:46.664965 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.664805 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-istio-data" (OuterVolumeSpecName: "istio-data") pod "5499e973-b228-4457-9073-b338ca8fa427" (UID: "5499e973-b228-4457-9073-b338ca8fa427"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:52:46.665078 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.665027 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5499e973-b228-4457-9073-b338ca8fa427" (UID: "5499e973-b228-4457-9073-b338ca8fa427"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:52:46.665078 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.665045 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5499e973-b228-4457-9073-b338ca8fa427" (UID: "5499e973-b228-4457-9073-b338ca8fa427"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:52:46.665186 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.665111 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-workload-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:52:46.665186 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.665126 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-istio-data\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:52:46.665259 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.665206 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5499e973-b228-4457-9073-b338ca8fa427-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5499e973-b228-4457-9073-b338ca8fa427" (UID: "5499e973-b228-4457-9073-b338ca8fa427"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:52:46.666787 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.666760 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5499e973-b228-4457-9073-b338ca8fa427" (UID: "5499e973-b228-4457-9073-b338ca8fa427"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:52:46.666895 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.666812 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5499e973-b228-4457-9073-b338ca8fa427-istio-token" (OuterVolumeSpecName: "istio-token") pod "5499e973-b228-4457-9073-b338ca8fa427" (UID: "5499e973-b228-4457-9073-b338ca8fa427"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:52:46.666986 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.666964 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5499e973-b228-4457-9073-b338ca8fa427-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5499e973-b228-4457-9073-b338ca8fa427" (UID: "5499e973-b228-4457-9073-b338ca8fa427"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 11:52:46.667049 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.666986 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5499e973-b228-4457-9073-b338ca8fa427-kube-api-access-vq8dn" (OuterVolumeSpecName: "kube-api-access-vq8dn") pod "5499e973-b228-4457-9073-b338ca8fa427" (UID: "5499e973-b228-4457-9073-b338ca8fa427"). InnerVolumeSpecName "kube-api-access-vq8dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:52:46.766232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.766201 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-credential-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:52:46.766232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.766227 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq8dn\" (UniqueName: \"kubernetes.io/projected/5499e973-b228-4457-9073-b338ca8fa427-kube-api-access-vq8dn\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:52:46.766428 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.766241 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5499e973-b228-4457-9073-b338ca8fa427-istio-podinfo\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:52:46.766428 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.766256 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-istio-envoy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:52:46.766428 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.766267 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5499e973-b228-4457-9073-b338ca8fa427-istiod-ca-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:52:46.766428 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.766278 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5499e973-b228-4457-9073-b338ca8fa427-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:52:46.766428 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:46.766289 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5499e973-b228-4457-9073-b338ca8fa427-workload-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:52:47.233729 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:47.233694 2564 generic.go:356] "Generic (PLEG): container finished" podID="5499e973-b228-4457-9073-b338ca8fa427" containerID="3746bf53ffbc4373240b55243bf87753a2985fc366a16081b69a0b85ebe4736a" exitCode=0 Sep 08 11:52:47.233902 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:47.233743 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" Sep 08 11:52:47.233902 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:47.233783 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" event={"ID":"5499e973-b228-4457-9073-b338ca8fa427","Type":"ContainerDied","Data":"3746bf53ffbc4373240b55243bf87753a2985fc366a16081b69a0b85ebe4736a"} Sep 08 11:52:47.233902 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:47.233846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs" event={"ID":"5499e973-b228-4457-9073-b338ca8fa427","Type":"ContainerDied","Data":"fabb82e88ea9f431a05507bedbae37a116203d7fa95f805063fcddedcb0a32b1"} Sep 08 11:52:47.233902 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:47.233876 2564 scope.go:117] "RemoveContainer" containerID="3746bf53ffbc4373240b55243bf87753a2985fc366a16081b69a0b85ebe4736a" Sep 08 11:52:47.244449 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:47.244434 2564 scope.go:117] "RemoveContainer" containerID="4c0b58c8c1598e4825d1963101ed22761ef6f27a06eae3449b4a03f01c453930" Sep 08 11:52:47.253096 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:47.253081 2564 scope.go:117] "RemoveContainer" containerID="3746bf53ffbc4373240b55243bf87753a2985fc366a16081b69a0b85ebe4736a" Sep 08 11:52:47.253359 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:52:47.253338 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3746bf53ffbc4373240b55243bf87753a2985fc366a16081b69a0b85ebe4736a\": container with ID starting with 3746bf53ffbc4373240b55243bf87753a2985fc366a16081b69a0b85ebe4736a not found: ID does not exist" containerID="3746bf53ffbc4373240b55243bf87753a2985fc366a16081b69a0b85ebe4736a" Sep 08 11:52:47.253487 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:47.253406 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3746bf53ffbc4373240b55243bf87753a2985fc366a16081b69a0b85ebe4736a"} err="failed to get container status \"3746bf53ffbc4373240b55243bf87753a2985fc366a16081b69a0b85ebe4736a\": rpc error: code = NotFound desc = could not find container \"3746bf53ffbc4373240b55243bf87753a2985fc366a16081b69a0b85ebe4736a\": container with ID starting with 3746bf53ffbc4373240b55243bf87753a2985fc366a16081b69a0b85ebe4736a not found: ID does not exist" Sep 08 11:52:47.253487 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:47.253432 2564 scope.go:117] "RemoveContainer" containerID="4c0b58c8c1598e4825d1963101ed22761ef6f27a06eae3449b4a03f01c453930" Sep 08 11:52:47.253714 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:52:47.253696 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0b58c8c1598e4825d1963101ed22761ef6f27a06eae3449b4a03f01c453930\": container with ID starting with 4c0b58c8c1598e4825d1963101ed22761ef6f27a06eae3449b4a03f01c453930 not found: ID does not exist" containerID="4c0b58c8c1598e4825d1963101ed22761ef6f27a06eae3449b4a03f01c453930" Sep 08 11:52:47.253753 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:47.253719 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0b58c8c1598e4825d1963101ed22761ef6f27a06eae3449b4a03f01c453930"} err="failed to get container status \"4c0b58c8c1598e4825d1963101ed22761ef6f27a06eae3449b4a03f01c453930\": rpc error: code = NotFound desc = could not find container \"4c0b58c8c1598e4825d1963101ed22761ef6f27a06eae3449b4a03f01c453930\": container with ID starting with 4c0b58c8c1598e4825d1963101ed22761ef6f27a06eae3449b4a03f01c453930 not found: ID does not exist" Sep 08 11:52:47.270975 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:47.270950 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs"] Sep 08 11:52:47.279861 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:47.279837 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-63d6ac92-openshift-default-5bcf557c5b-dn4gs"] Sep 08 11:52:47.654008 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:47.653969 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5499e973-b228-4457-9073-b338ca8fa427" path="/var/lib/kubelet/pods/5499e973-b228-4457-9073-b338ca8fa427/volumes" Sep 08 11:52:53.760384 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:53.760341 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:53.760778 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:53.760419 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:53.772995 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:53.772971 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:54.284007 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:54.283969 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:52:56.542157 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:52:56.542118 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:53:13.240967 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:13.240933 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:53:26.747866 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:26.747821 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:53:37.567102 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.567072 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch"] Sep 08 11:53:37.567516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.567502 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5499e973-b228-4457-9073-b338ca8fa427" containerName="istio-proxy" Sep 08 11:53:37.567569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.567518 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5499e973-b228-4457-9073-b338ca8fa427" containerName="istio-proxy" Sep 08 11:53:37.567569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.567536 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5499e973-b228-4457-9073-b338ca8fa427" containerName="istio-proxy" Sep 08 11:53:37.567569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.567541 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5499e973-b228-4457-9073-b338ca8fa427" containerName="istio-proxy" Sep 08 11:53:37.567663 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.567608 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5499e973-b228-4457-9073-b338ca8fa427" containerName="istio-proxy" Sep 08 11:53:37.567663 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.567617 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5499e973-b228-4457-9073-b338ca8fa427" containerName="istio-proxy" Sep 08 11:53:37.571991 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.571971 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.576802 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.576777 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-77fac0-openshift-default-dockercfg-xr5kc\"" Sep 08 11:53:37.627167 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.627138 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch"] Sep 08 11:53:37.654129 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.654083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f263488-35f0-4110-92b9-460e5ee4c449-istiod-ca-cert\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.654701 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.654675 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-istio-data\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.654831 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.654728 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f263488-35f0-4110-92b9-460e5ee4c449-istio-token\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.654888 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.654827 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-workload-certs\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.654946 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.654888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-workload-socket\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.655005 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.654952 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-istio-envoy\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.655080 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.655039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djsc9\" (UniqueName: \"kubernetes.io/projected/5f263488-35f0-4110-92b9-460e5ee4c449-kube-api-access-djsc9\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.655080 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.655070 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f263488-35f0-4110-92b9-460e5ee4c449-istio-podinfo\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.655202 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.655117 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-credential-socket\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.756196 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.756154 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-credential-socket\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.756418 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.756316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f263488-35f0-4110-92b9-460e5ee4c449-istiod-ca-cert\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.756418 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.756394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-istio-data\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.756555 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.756457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f263488-35f0-4110-92b9-460e5ee4c449-istio-token\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.756555 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.756542 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-workload-certs\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.756660 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.756581 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-workload-socket\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.756660 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.756628 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-credential-socket\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.756660 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.756635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-istio-envoy\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.756816 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.756706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djsc9\" (UniqueName: \"kubernetes.io/projected/5f263488-35f0-4110-92b9-460e5ee4c449-kube-api-access-djsc9\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.756816 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.756737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f263488-35f0-4110-92b9-460e5ee4c449-istio-podinfo\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.757038 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.756970 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-istio-data\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.757038 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.757000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-workload-certs\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.757159 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.757077 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-workload-socket\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.757389 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.757346 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f263488-35f0-4110-92b9-460e5ee4c449-istiod-ca-cert\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.759017 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.758986 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-istio-envoy\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.759160 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.759142 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f263488-35f0-4110-92b9-460e5ee4c449-istio-podinfo\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.765263 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.765239 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f263488-35f0-4110-92b9-460e5ee4c449-istio-token\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.765580 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.765558 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djsc9\" (UniqueName: \"kubernetes.io/projected/5f263488-35f0-4110-92b9-460e5ee4c449-kube-api-access-djsc9\") pod \"e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:37.882657 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:37.882557 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:38.161891 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:38.161860 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch"] Sep 08 11:53:38.162850 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:53:38.162826 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f263488_35f0_4110_92b9_460e5ee4c449.slice/crio-e2a74358cce06106aad3a80ab9804d7152c38e61b4b98b9e1b741e4f91014590 WatchSource:0}: Error finding container e2a74358cce06106aad3a80ab9804d7152c38e61b4b98b9e1b741e4f91014590: Status 404 returned error can't find the container with id e2a74358cce06106aad3a80ab9804d7152c38e61b4b98b9e1b741e4f91014590 Sep 08 11:53:38.165303 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:38.165258 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:53:38.165401 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:38.165356 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:53:38.165442 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:38.165420 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:53:38.534481 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:38.534381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" event={"ID":"5f263488-35f0-4110-92b9-460e5ee4c449","Type":"ContainerStarted","Data":"ec53031833862c545be600c8ac049b10ce5008258c112fbeab52a871def12346"} Sep 08 11:53:38.534481 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:38.534445 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" event={"ID":"5f263488-35f0-4110-92b9-460e5ee4c449","Type":"ContainerStarted","Data":"e2a74358cce06106aad3a80ab9804d7152c38e61b4b98b9e1b741e4f91014590"} Sep 08 11:53:38.561360 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:38.561305 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" podStartSLOduration=1.5612904570000001 podStartE2EDuration="1.561290457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:53:38.559839363 +0000 UTC m=+1407.419026703" watchObservedRunningTime="2026-09-08 11:53:38.561290457 +0000 UTC m=+1407.420477772" Sep 08 11:53:38.883714 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:38.883624 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:39.888096 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:39.888064 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:40.545577 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:40.545547 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:40.546441 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:40.546421 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:41.597211 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.597178 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4"] Sep 08 11:53:41.600907 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.600890 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:41.603734 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.603713 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-77fac0-serving-cert\"" Sep 08 11:53:41.616170 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.616144 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4"] Sep 08 11:53:41.695032 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.694988 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa47f559-e025-43bd-9867-28b56a844ada-openshift-service-ca\") pod \"maas-api-e2e-api-a-77fac0-689d946ffc-648g4\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:41.695032 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.695034 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa47f559-e025-43bd-9867-28b56a844ada-metrics-certs\") pod \"maas-api-e2e-api-a-77fac0-689d946ffc-648g4\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:41.695282 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.695137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndr9f\" (UniqueName: \"kubernetes.io/projected/fa47f559-e025-43bd-9867-28b56a844ada-kube-api-access-ndr9f\") pod \"maas-api-e2e-api-a-77fac0-689d946ffc-648g4\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:41.695282 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.695187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa47f559-e025-43bd-9867-28b56a844ada-maas-api-tls\") pod \"maas-api-e2e-api-a-77fac0-689d946ffc-648g4\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:41.795706 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.795667 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa47f559-e025-43bd-9867-28b56a844ada-openshift-service-ca\") pod \"maas-api-e2e-api-a-77fac0-689d946ffc-648g4\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:41.795706 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.795708 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa47f559-e025-43bd-9867-28b56a844ada-metrics-certs\") pod \"maas-api-e2e-api-a-77fac0-689d946ffc-648g4\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:41.795943 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.795784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndr9f\" (UniqueName: \"kubernetes.io/projected/fa47f559-e025-43bd-9867-28b56a844ada-kube-api-access-ndr9f\") pod \"maas-api-e2e-api-a-77fac0-689d946ffc-648g4\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:41.795943 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.795822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa47f559-e025-43bd-9867-28b56a844ada-maas-api-tls\") pod \"maas-api-e2e-api-a-77fac0-689d946ffc-648g4\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:41.796491 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.796465 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa47f559-e025-43bd-9867-28b56a844ada-openshift-service-ca\") pod \"maas-api-e2e-api-a-77fac0-689d946ffc-648g4\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:41.798275 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.798243 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa47f559-e025-43bd-9867-28b56a844ada-metrics-certs\") pod \"maas-api-e2e-api-a-77fac0-689d946ffc-648g4\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:41.798275 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.798267 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa47f559-e025-43bd-9867-28b56a844ada-maas-api-tls\") pod \"maas-api-e2e-api-a-77fac0-689d946ffc-648g4\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:41.808525 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.808499 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndr9f\" (UniqueName: \"kubernetes.io/projected/fa47f559-e025-43bd-9867-28b56a844ada-kube-api-access-ndr9f\") pod \"maas-api-e2e-api-a-77fac0-689d946ffc-648g4\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:41.913816 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:41.913778 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:42.192872 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:42.192842 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4"] Sep 08 11:53:42.193657 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:53:42.193629 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa47f559_e025_43bd_9867_28b56a844ada.slice/crio-d43310a89007db49ff2cee8fe1089bdf61cab67d9088afb1c2d1f6daf0e7558d WatchSource:0}: Error finding container d43310a89007db49ff2cee8fe1089bdf61cab67d9088afb1c2d1f6daf0e7558d: Status 404 returned error can't find the container with id d43310a89007db49ff2cee8fe1089bdf61cab67d9088afb1c2d1f6daf0e7558d Sep 08 11:53:42.195177 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:42.195161 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 11:53:42.560656 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:42.560580 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" event={"ID":"fa47f559-e025-43bd-9867-28b56a844ada","Type":"ContainerStarted","Data":"d43310a89007db49ff2cee8fe1089bdf61cab67d9088afb1c2d1f6daf0e7558d"} Sep 08 11:53:43.568717 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:43.568671 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" event={"ID":"fa47f559-e025-43bd-9867-28b56a844ada","Type":"ContainerStarted","Data":"41d9e1e464e8bfe71f91112f777919f6a6014e40e6ebbaa25f15a143cb407603"} Sep 08 11:53:43.569172 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:43.568737 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:43.591722 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:43.591671 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" podStartSLOduration=2.160487068 podStartE2EDuration="2.591655661s" totalImagesPullingTime="431.168593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:53:43.590591585 +0000 UTC m=+1412.449778919" watchObservedRunningTime="2026-09-08 11:53:43.591655661 +0000 UTC m=+1412.450842979" Sep 08 11:53:46.550248 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:46.550207 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:53:49.591298 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:49.591255 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:53:50.327413 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:50.327379 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f"] Sep 08 11:53:56.305968 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.305930 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w"] Sep 08 11:53:56.309979 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.309954 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:56.315049 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.315027 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3f9288-serving-cert\"" Sep 08 11:53:56.328337 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.328309 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w"] Sep 08 11:53:56.356572 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.356537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgmbw\" (UniqueName: \"kubernetes.io/projected/daad897f-a8af-449f-9176-998ff78f2535-kube-api-access-qgmbw\") pod \"maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:56.356753 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.356589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad897f-a8af-449f-9176-998ff78f2535-metrics-certs\") pod \"maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:56.356753 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.356664 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daad897f-a8af-449f-9176-998ff78f2535-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:56.356753 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.356703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daad897f-a8af-449f-9176-998ff78f2535-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:56.457640 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.457599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daad897f-a8af-449f-9176-998ff78f2535-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:56.457843 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.457689 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgmbw\" (UniqueName: \"kubernetes.io/projected/daad897f-a8af-449f-9176-998ff78f2535-kube-api-access-qgmbw\") pod \"maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:56.457843 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.457728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad897f-a8af-449f-9176-998ff78f2535-metrics-certs\") pod \"maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:56.457843 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.457772 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daad897f-a8af-449f-9176-998ff78f2535-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:56.457843 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:53:56.457773 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-serving-cert: secret "maas-api-e2e-ipp-a-3f9288-serving-cert" not found Sep 08 11:53:56.458063 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:53:56.457851 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daad897f-a8af-449f-9176-998ff78f2535-maas-api-tls podName:daad897f-a8af-449f-9176-998ff78f2535 nodeName:}" failed. No retries permitted until 2026-09-08 11:53:56.957829738 +0000 UTC m=+1425.817017034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/daad897f-a8af-449f-9176-998ff78f2535-maas-api-tls") pod "maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" (UID: "daad897f-a8af-449f-9176-998ff78f2535") : secret "maas-api-e2e-ipp-a-3f9288-serving-cert" not found Sep 08 11:53:56.458520 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.458502 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daad897f-a8af-449f-9176-998ff78f2535-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:56.460378 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.460341 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad897f-a8af-449f-9176-998ff78f2535-metrics-certs\") pod \"maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:56.466624 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.466600 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgmbw\" (UniqueName: \"kubernetes.io/projected/daad897f-a8af-449f-9176-998ff78f2535-kube-api-access-qgmbw\") pod \"maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:56.586570 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.586484 2564 patch_prober.go:28] interesting pod/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 11:53:56.586711 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.586572 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 11:53:56.963196 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.963111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daad897f-a8af-449f-9176-998ff78f2535-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:56.965719 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:56.965688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daad897f-a8af-449f-9176-998ff78f2535-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:57.221956 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:57.221856 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:57.495527 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:57.495499 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w"] Sep 08 11:53:57.496639 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:53:57.496610 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaad897f_a8af_449f_9176_998ff78f2535.slice/crio-93e5165a0a65cd9fb46d071f5bc5035177c9b34230f39a689df57147f8ca30a9 WatchSource:0}: Error finding container 93e5165a0a65cd9fb46d071f5bc5035177c9b34230f39a689df57147f8ca30a9: Status 404 returned error can't find the container with id 93e5165a0a65cd9fb46d071f5bc5035177c9b34230f39a689df57147f8ca30a9 Sep 08 11:53:57.662845 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:57.662813 2564 generic.go:356] "Generic (PLEG): container finished" podID="5f263488-35f0-4110-92b9-460e5ee4c449" containerID="ec53031833862c545be600c8ac049b10ce5008258c112fbeab52a871def12346" exitCode=0 Sep 08 11:53:57.663035 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:57.662891 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" event={"ID":"5f263488-35f0-4110-92b9-460e5ee4c449","Type":"ContainerDied","Data":"ec53031833862c545be600c8ac049b10ce5008258c112fbeab52a871def12346"} Sep 08 11:53:57.663463 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:57.663445 2564 scope.go:117] "RemoveContainer" containerID="ec53031833862c545be600c8ac049b10ce5008258c112fbeab52a871def12346" Sep 08 11:53:57.664242 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:57.664220 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" event={"ID":"daad897f-a8af-449f-9176-998ff78f2535","Type":"ContainerStarted","Data":"93e5165a0a65cd9fb46d071f5bc5035177c9b34230f39a689df57147f8ca30a9"} Sep 08 11:53:57.664600 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:57.664572 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:53:57.664655 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:57.664643 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:53:57.664707 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:57.664676 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:53:58.672426 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:58.672394 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" event={"ID":"5f263488-35f0-4110-92b9-460e5ee4c449","Type":"ContainerStarted","Data":"52fcc2b420cc11223c5de1ece1a8b541f48a31c30be18049d5530a3f226ed6fe"} Sep 08 11:53:58.674302 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:58.674256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" event={"ID":"daad897f-a8af-449f-9176-998ff78f2535","Type":"ContainerStarted","Data":"a54b5304db1b8589fbd55bd76fa524cc8cf940fff5f528f7b86f2f62a7df18b3"} Sep 08 11:53:58.674649 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:58.674626 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:53:58.720138 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:58.720076 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" podStartSLOduration=2.046369285 podStartE2EDuration="2.720060028s" totalImagesPullingTime="673.690743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:53:58.717857006 +0000 UTC m=+1427.577044345" watchObservedRunningTime="2026-09-08 11:53:58.720060028 +0000 UTC m=+1427.579247409" Sep 08 11:53:58.884070 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:58.884024 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:53:58.884301 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:58.884272 2564 patch_prober.go:28] interesting pod/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Sep 08 11:53:58.884399 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:58.884349 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Sep 08 11:53:59.884001 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:59.883954 2564 patch_prober.go:28] interesting pod/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Sep 08 11:53:59.884612 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:53:59.884577 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Sep 08 11:54:00.883968 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:00.883525 2564 patch_prober.go:28] interesting pod/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Sep 08 11:54:00.883968 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:00.883591 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Sep 08 11:54:01.883319 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:01.883265 2564 patch_prober.go:28] interesting pod/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Sep 08 11:54:01.883556 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:01.883343 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Sep 08 11:54:02.883567 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:02.883524 2564 patch_prober.go:28] interesting pod/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Sep 08 11:54:02.884069 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:02.883600 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Sep 08 11:54:03.883037 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:03.882994 2564 patch_prober.go:28] interesting pod/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Sep 08 11:54:03.883212 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:03.883089 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Sep 08 11:54:04.690757 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:04.690730 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:54:04.887487 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:04.887457 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:54:04.887876 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:04.887847 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:54:04.888492 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:04.888472 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:54:05.347538 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.347501 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" podUID="5db1c580-2330-4820-b4df-f0b86b55330f" containerName="main" containerID="cri-o://e3467264c5196f459e4ea5b22eb1324442b5f0e658b4d00465e381ccfe2b98b2" gracePeriod=45 Sep 08 11:54:05.609188 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.609123 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:54:05.739539 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.739507 2564 generic.go:356] "Generic (PLEG): container finished" podID="5db1c580-2330-4820-b4df-f0b86b55330f" containerID="e3467264c5196f459e4ea5b22eb1324442b5f0e658b4d00465e381ccfe2b98b2" exitCode=0 Sep 08 11:54:05.740016 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.739548 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" Sep 08 11:54:05.740016 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.739591 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" event={"ID":"5db1c580-2330-4820-b4df-f0b86b55330f","Type":"ContainerDied","Data":"e3467264c5196f459e4ea5b22eb1324442b5f0e658b4d00465e381ccfe2b98b2"} Sep 08 11:54:05.740016 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.739637 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f" event={"ID":"5db1c580-2330-4820-b4df-f0b86b55330f","Type":"ContainerDied","Data":"8226e8d83e3eb306605d0075f9db0dc05bb570cd846b2c3060f497a6c02d272b"} Sep 08 11:54:05.740016 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.739664 2564 scope.go:117] "RemoveContainer" containerID="e3467264c5196f459e4ea5b22eb1324442b5f0e658b4d00465e381ccfe2b98b2" Sep 08 11:54:05.748869 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.748846 2564 scope.go:117] "RemoveContainer" containerID="e3467264c5196f459e4ea5b22eb1324442b5f0e658b4d00465e381ccfe2b98b2" Sep 08 11:54:05.749119 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:54:05.749101 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3467264c5196f459e4ea5b22eb1324442b5f0e658b4d00465e381ccfe2b98b2\": container with ID starting with e3467264c5196f459e4ea5b22eb1324442b5f0e658b4d00465e381ccfe2b98b2 not found: ID does not exist" containerID="e3467264c5196f459e4ea5b22eb1324442b5f0e658b4d00465e381ccfe2b98b2" Sep 08 11:54:05.749168 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.749128 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3467264c5196f459e4ea5b22eb1324442b5f0e658b4d00465e381ccfe2b98b2"} err="failed to get container status \"e3467264c5196f459e4ea5b22eb1324442b5f0e658b4d00465e381ccfe2b98b2\": rpc error: code = NotFound desc = could not find container \"e3467264c5196f459e4ea5b22eb1324442b5f0e658b4d00465e381ccfe2b98b2\": container with ID starting with e3467264c5196f459e4ea5b22eb1324442b5f0e658b4d00465e381ccfe2b98b2 not found: ID does not exist" Sep 08 11:54:05.758476 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.758457 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgzpm\" (UniqueName: \"kubernetes.io/projected/5db1c580-2330-4820-b4df-f0b86b55330f-kube-api-access-xgzpm\") pod \"5db1c580-2330-4820-b4df-f0b86b55330f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " Sep 08 11:54:05.758534 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.758502 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-home\") pod \"5db1c580-2330-4820-b4df-f0b86b55330f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " Sep 08 11:54:05.758581 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.758534 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-model-cache\") pod \"5db1c580-2330-4820-b4df-f0b86b55330f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " Sep 08 11:54:05.758581 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.758552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-tmp-dir\") pod \"5db1c580-2330-4820-b4df-f0b86b55330f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " Sep 08 11:54:05.758671 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.758611 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-dshm\") pod \"5db1c580-2330-4820-b4df-f0b86b55330f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " Sep 08 11:54:05.758671 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.758649 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5db1c580-2330-4820-b4df-f0b86b55330f-tls-certs\") pod \"5db1c580-2330-4820-b4df-f0b86b55330f\" (UID: \"5db1c580-2330-4820-b4df-f0b86b55330f\") " Sep 08 11:54:05.758826 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.758801 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-model-cache" (OuterVolumeSpecName: "model-cache") pod "5db1c580-2330-4820-b4df-f0b86b55330f" (UID: "5db1c580-2330-4820-b4df-f0b86b55330f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:54:05.758918 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.758812 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-home" (OuterVolumeSpecName: "home") pod "5db1c580-2330-4820-b4df-f0b86b55330f" (UID: "5db1c580-2330-4820-b4df-f0b86b55330f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:54:05.758918 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.758827 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5db1c580-2330-4820-b4df-f0b86b55330f" (UID: "5db1c580-2330-4820-b4df-f0b86b55330f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:54:05.759042 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.758996 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-home\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:54:05.759042 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.759020 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-model-cache\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:54:05.759042 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.759035 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-tmp-dir\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:54:05.760701 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.760675 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-dshm" (OuterVolumeSpecName: "dshm") pod "5db1c580-2330-4820-b4df-f0b86b55330f" (UID: "5db1c580-2330-4820-b4df-f0b86b55330f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:54:05.760793 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.760766 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db1c580-2330-4820-b4df-f0b86b55330f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5db1c580-2330-4820-b4df-f0b86b55330f" (UID: "5db1c580-2330-4820-b4df-f0b86b55330f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:54:05.760841 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.760818 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db1c580-2330-4820-b4df-f0b86b55330f-kube-api-access-xgzpm" (OuterVolumeSpecName: "kube-api-access-xgzpm") pod "5db1c580-2330-4820-b4df-f0b86b55330f" (UID: "5db1c580-2330-4820-b4df-f0b86b55330f"). InnerVolumeSpecName "kube-api-access-xgzpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:54:05.860018 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.859925 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5db1c580-2330-4820-b4df-f0b86b55330f-dshm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:54:05.860018 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.859953 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5db1c580-2330-4820-b4df-f0b86b55330f-tls-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:54:05.860018 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:05.859966 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgzpm\" (UniqueName: \"kubernetes.io/projected/5db1c580-2330-4820-b4df-f0b86b55330f-kube-api-access-xgzpm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:54:06.065207 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.065178 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f"] Sep 08 11:54:06.069033 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.069005 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-838a68/rate-test-model-838a68-kserve-6d744c5fcf-4qt8f"] Sep 08 11:54:06.209624 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.209578 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr"] Sep 08 11:54:06.210079 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.210061 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db1c580-2330-4820-b4df-f0b86b55330f" containerName="main" Sep 08 11:54:06.210155 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.210084 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db1c580-2330-4820-b4df-f0b86b55330f" containerName="main" Sep 08 11:54:06.210256 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.210229 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db1c580-2330-4820-b4df-f0b86b55330f" containerName="main" Sep 08 11:54:06.213278 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.213262 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.216804 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.216780 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ca8280-openshift-default-dockercfg-gkpmf\"" Sep 08 11:54:06.229584 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.229559 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr"] Sep 08 11:54:06.365747 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.365710 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-workload-socket\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.365900 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.365808 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istiod-ca-cert\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.365900 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.365866 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-data\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.365991 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.365913 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-podinfo\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.365991 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.365983 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-token\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.366067 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.366010 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vpmv\" (UniqueName: \"kubernetes.io/projected/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-kube-api-access-6vpmv\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.366118 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.366087 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-envoy\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.366237 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.366217 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-credential-socket\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.366289 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.366264 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-workload-certs\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.467404 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.467301 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-envoy\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.467404 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.467389 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-credential-socket\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.467632 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.467416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-workload-certs\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.467632 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.467434 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-workload-socket\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.467632 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.467464 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istiod-ca-cert\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.467632 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.467498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-data\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.467632 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.467545 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-podinfo\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.467632 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.467593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-token\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.467632 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.467614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vpmv\" (UniqueName: \"kubernetes.io/projected/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-kube-api-access-6vpmv\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.467989 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.467874 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-workload-socket\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.467989 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.467903 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-workload-certs\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.468313 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.468282 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-credential-socket\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.468441 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.468282 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istiod-ca-cert\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.468441 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.468289 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-data\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.469703 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.469672 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-envoy\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.469953 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.469934 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-podinfo\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.478127 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.478104 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vpmv\" (UniqueName: \"kubernetes.io/projected/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-kube-api-access-6vpmv\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.478745 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.478727 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-token\") pod \"e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.524232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.524201 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:06.804331 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.804295 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr"] Sep 08 11:54:06.805423 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:54:06.805398 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb81f47f_e110_4d24_a80d_cc2ada6a5e3a.slice/crio-861c35341175d827a5637bc667aa49da515e4a2ac9fc2588438ef03a68524c61 WatchSource:0}: Error finding container 861c35341175d827a5637bc667aa49da515e4a2ac9fc2588438ef03a68524c61: Status 404 returned error can't find the container with id 861c35341175d827a5637bc667aa49da515e4a2ac9fc2588438ef03a68524c61 Sep 08 11:54:06.807719 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.807688 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:54:06.807782 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.807749 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:54:06.807782 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:06.807778 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:54:07.653233 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:07.653197 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5db1c580-2330-4820-b4df-f0b86b55330f" path="/var/lib/kubelet/pods/5db1c580-2330-4820-b4df-f0b86b55330f/volumes" Sep 08 11:54:07.756179 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:07.756145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" event={"ID":"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a","Type":"ContainerStarted","Data":"5f9f9cff6dffcb9ce108b067c0838235950097e0d1f021175d9f8b6f7db47090"} Sep 08 11:54:07.756344 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:07.756201 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" event={"ID":"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a","Type":"ContainerStarted","Data":"861c35341175d827a5637bc667aa49da515e4a2ac9fc2588438ef03a68524c61"} Sep 08 11:54:07.780725 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:07.780683 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" podStartSLOduration=1.7806688739999998 podStartE2EDuration="1.780668874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:54:07.779826903 +0000 UTC m=+1436.639014248" watchObservedRunningTime="2026-09-08 11:54:07.780668874 +0000 UTC m=+1436.639856191" Sep 08 11:54:08.525269 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:08.525228 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:08.530716 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:08.530688 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:08.762150 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:08.762120 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:08.763044 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:08.763026 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:10.826225 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:10.826191 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb"] Sep 08 11:54:10.829685 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:10.829666 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:10.832304 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:10.832280 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ca8280-serving-cert\"" Sep 08 11:54:10.842529 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:10.842502 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb"] Sep 08 11:54:10.917153 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:10.917119 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/947f2822-1b50-432f-9d97-9e077cf0d554-openshift-service-ca\") pod \"maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:10.917311 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:10.917204 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndrdp\" (UniqueName: \"kubernetes.io/projected/947f2822-1b50-432f-9d97-9e077cf0d554-kube-api-access-ndrdp\") pod \"maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:10.917311 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:10.917230 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/947f2822-1b50-432f-9d97-9e077cf0d554-maas-api-tls\") pod \"maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:10.917509 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:10.917483 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/947f2822-1b50-432f-9d97-9e077cf0d554-metrics-certs\") pod \"maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:11.018800 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:11.018763 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/947f2822-1b50-432f-9d97-9e077cf0d554-metrics-certs\") pod \"maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:11.018975 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:11.018836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/947f2822-1b50-432f-9d97-9e077cf0d554-openshift-service-ca\") pod \"maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:11.018975 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:11.018937 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndrdp\" (UniqueName: \"kubernetes.io/projected/947f2822-1b50-432f-9d97-9e077cf0d554-kube-api-access-ndrdp\") pod \"maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:11.019123 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:11.018971 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/947f2822-1b50-432f-9d97-9e077cf0d554-maas-api-tls\") pod \"maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:11.019587 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:11.019560 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/947f2822-1b50-432f-9d97-9e077cf0d554-openshift-service-ca\") pod \"maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:11.021243 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:11.021223 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/947f2822-1b50-432f-9d97-9e077cf0d554-metrics-certs\") pod \"maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:11.021363 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:11.021268 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/947f2822-1b50-432f-9d97-9e077cf0d554-maas-api-tls\") pod \"maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:11.030191 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:11.030170 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndrdp\" (UniqueName: \"kubernetes.io/projected/947f2822-1b50-432f-9d97-9e077cf0d554-kube-api-access-ndrdp\") pod \"maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:11.143313 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:11.143275 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:11.422362 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:11.422339 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb"] Sep 08 11:54:11.427324 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:54:11.427293 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947f2822_1b50_432f_9d97_9e077cf0d554.slice/crio-e978752380c52968fed3fb3570f755e4e45bc7aa2f56e0054bf23e5125522b94 WatchSource:0}: Error finding container e978752380c52968fed3fb3570f755e4e45bc7aa2f56e0054bf23e5125522b94: Status 404 returned error can't find the container with id e978752380c52968fed3fb3570f755e4e45bc7aa2f56e0054bf23e5125522b94 Sep 08 11:54:11.782412 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:11.782381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" event={"ID":"947f2822-1b50-432f-9d97-9e077cf0d554","Type":"ContainerStarted","Data":"e978752380c52968fed3fb3570f755e4e45bc7aa2f56e0054bf23e5125522b94"} Sep 08 11:54:12.043258 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:12.043184 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:54:12.791429 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:12.791392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" event={"ID":"947f2822-1b50-432f-9d97-9e077cf0d554","Type":"ContainerStarted","Data":"72b3d6ccde4f71102edc8e74a214d7c5cebc69136f319dc9805be2887dc77c44"} Sep 08 11:54:12.791601 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:12.791504 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:12.810595 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:12.810543 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" podStartSLOduration=2.504734008 podStartE2EDuration="2.810529605s" totalImagesPullingTime="305.795597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:54:12.80949667 +0000 UTC m=+1441.668684002" watchObservedRunningTime="2026-09-08 11:54:12.810529605 +0000 UTC m=+1441.669716923" Sep 08 11:54:18.804404 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:18.804348 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:54:25.408459 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.408426 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25"] Sep 08 11:54:25.413591 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.413566 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:25.418404 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.418334 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2ebeba-serving-cert\"" Sep 08 11:54:25.427264 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.427234 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25"] Sep 08 11:54:25.464943 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.464912 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:25.465084 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.464960 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcbgc\" (UniqueName: \"kubernetes.io/projected/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-kube-api-access-fcbgc\") pod \"maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:25.465084 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.465017 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-metrics-certs\") pod \"maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:25.465159 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.465089 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:25.565800 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.565768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:25.565969 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.565822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcbgc\" (UniqueName: \"kubernetes.io/projected/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-kube-api-access-fcbgc\") pod \"maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:25.565969 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.565854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-metrics-certs\") pod \"maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:25.565969 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.565876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:25.566522 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.566484 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:25.568382 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.568355 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-metrics-certs\") pod \"maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:25.568515 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.568356 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:25.576390 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.576345 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcbgc\" (UniqueName: \"kubernetes.io/projected/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-kube-api-access-fcbgc\") pod \"maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:25.729093 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:25.729012 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:26.041354 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:26.041321 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25"] Sep 08 11:54:26.041493 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:54:26.041354 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe0571a_9e6f_4d0a_9f5f_1e274f7ad4f3.slice/crio-d427c60993a6f1c6968177264b590ad8993424aa8c209397ac1006b6ad65ea40 WatchSource:0}: Error finding container d427c60993a6f1c6968177264b590ad8993424aa8c209397ac1006b6ad65ea40: Status 404 returned error can't find the container with id d427c60993a6f1c6968177264b590ad8993424aa8c209397ac1006b6ad65ea40 Sep 08 11:54:26.908107 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:26.908064 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" event={"ID":"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3","Type":"ContainerStarted","Data":"5b662c52274c675090feaa8490960af7735c6b024d98c331603e0040cf5c6413"} Sep 08 11:54:26.908523 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:26.908130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" event={"ID":"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3","Type":"ContainerStarted","Data":"d427c60993a6f1c6968177264b590ad8993424aa8c209397ac1006b6ad65ea40"} Sep 08 11:54:26.908523 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:26.908161 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:26.930436 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:26.930387 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" podStartSLOduration=1.616672084 podStartE2EDuration="1.930357263s" totalImagesPullingTime="313.685179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:54:26.928735229 +0000 UTC m=+1455.787922567" watchObservedRunningTime="2026-09-08 11:54:26.930357263 +0000 UTC m=+1455.789544581" Sep 08 11:54:31.550385 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:31.550329 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:54:31.946982 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:31.946949 2564 generic.go:356] "Generic (PLEG): container finished" podID="bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" containerID="5f9f9cff6dffcb9ce108b067c0838235950097e0d1f021175d9f8b6f7db47090" exitCode=0 Sep 08 11:54:31.947161 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:31.947025 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" event={"ID":"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a","Type":"ContainerDied","Data":"5f9f9cff6dffcb9ce108b067c0838235950097e0d1f021175d9f8b6f7db47090"} Sep 08 11:54:31.947513 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:31.947499 2564 scope.go:117] "RemoveContainer" containerID="5f9f9cff6dffcb9ce108b067c0838235950097e0d1f021175d9f8b6f7db47090" Sep 08 11:54:31.948810 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:31.948783 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:54:31.948881 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:31.948867 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:54:31.948921 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:31.948903 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:54:32.921773 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:32.921747 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:54:32.956831 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:32.956796 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" event={"ID":"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a","Type":"ContainerStarted","Data":"d2b17197429769eba2725346818c9b3f943986503f25d1e9a48741c1996cc3f6"} Sep 08 11:54:33.525036 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:33.524998 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:33.525233 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:33.525197 2564 patch_prober.go:28] interesting pod/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 08 11:54:33.525316 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:33.525267 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" podUID="bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 08 11:54:34.524878 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:34.524841 2564 patch_prober.go:28] interesting pod/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 08 11:54:34.525273 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:34.524912 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" podUID="bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 08 11:54:35.525348 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:35.525311 2564 patch_prober.go:28] interesting pod/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 08 11:54:35.525734 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:35.525388 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" podUID="bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 08 11:54:36.525124 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:36.525083 2564 patch_prober.go:28] interesting pod/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 08 11:54:36.525336 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:36.525162 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" podUID="bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 08 11:54:37.524670 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:37.524636 2564 patch_prober.go:28] interesting pod/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 08 11:54:37.525135 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:37.524694 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" podUID="bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 08 11:54:38.529706 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:38.529673 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:38.530097 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:38.530056 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:38.530670 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:38.530648 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:54:52.679467 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:52.679434 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4"] Sep 08 11:54:52.680319 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:52.679640 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" podUID="fa47f559-e025-43bd-9867-28b56a844ada" containerName="maas-api" containerID="cri-o://41d9e1e464e8bfe71f91112f777919f6a6014e40e6ebbaa25f15a143cb407603" gracePeriod=30 Sep 08 11:54:52.932427 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:52.932349 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:54:53.055355 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.055322 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa47f559-e025-43bd-9867-28b56a844ada-metrics-certs\") pod \"fa47f559-e025-43bd-9867-28b56a844ada\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " Sep 08 11:54:53.055532 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.055362 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa47f559-e025-43bd-9867-28b56a844ada-openshift-service-ca\") pod \"fa47f559-e025-43bd-9867-28b56a844ada\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " Sep 08 11:54:53.055532 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.055448 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa47f559-e025-43bd-9867-28b56a844ada-maas-api-tls\") pod \"fa47f559-e025-43bd-9867-28b56a844ada\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " Sep 08 11:54:53.055652 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.055552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndr9f\" (UniqueName: \"kubernetes.io/projected/fa47f559-e025-43bd-9867-28b56a844ada-kube-api-access-ndr9f\") pod \"fa47f559-e025-43bd-9867-28b56a844ada\" (UID: \"fa47f559-e025-43bd-9867-28b56a844ada\") " Sep 08 11:54:53.055850 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.055812 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa47f559-e025-43bd-9867-28b56a844ada-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fa47f559-e025-43bd-9867-28b56a844ada" (UID: "fa47f559-e025-43bd-9867-28b56a844ada"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:54:53.057548 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.057513 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa47f559-e025-43bd-9867-28b56a844ada-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fa47f559-e025-43bd-9867-28b56a844ada" (UID: "fa47f559-e025-43bd-9867-28b56a844ada"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:54:53.057675 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.057568 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa47f559-e025-43bd-9867-28b56a844ada-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fa47f559-e025-43bd-9867-28b56a844ada" (UID: "fa47f559-e025-43bd-9867-28b56a844ada"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:54:53.057675 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.057586 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa47f559-e025-43bd-9867-28b56a844ada-kube-api-access-ndr9f" (OuterVolumeSpecName: "kube-api-access-ndr9f") pod "fa47f559-e025-43bd-9867-28b56a844ada" (UID: "fa47f559-e025-43bd-9867-28b56a844ada"). InnerVolumeSpecName "kube-api-access-ndr9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:54:53.110554 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.110505 2564 generic.go:356] "Generic (PLEG): container finished" podID="fa47f559-e025-43bd-9867-28b56a844ada" containerID="41d9e1e464e8bfe71f91112f777919f6a6014e40e6ebbaa25f15a143cb407603" exitCode=0 Sep 08 11:54:53.110554 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.110560 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" Sep 08 11:54:53.110839 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.110637 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" event={"ID":"fa47f559-e025-43bd-9867-28b56a844ada","Type":"ContainerDied","Data":"41d9e1e464e8bfe71f91112f777919f6a6014e40e6ebbaa25f15a143cb407603"} Sep 08 11:54:53.110839 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.110730 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4" event={"ID":"fa47f559-e025-43bd-9867-28b56a844ada","Type":"ContainerDied","Data":"d43310a89007db49ff2cee8fe1089bdf61cab67d9088afb1c2d1f6daf0e7558d"} Sep 08 11:54:53.110839 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.110791 2564 scope.go:117] "RemoveContainer" containerID="41d9e1e464e8bfe71f91112f777919f6a6014e40e6ebbaa25f15a143cb407603" Sep 08 11:54:53.123523 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.123495 2564 scope.go:117] "RemoveContainer" containerID="41d9e1e464e8bfe71f91112f777919f6a6014e40e6ebbaa25f15a143cb407603" Sep 08 11:54:53.123841 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:54:53.123812 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d9e1e464e8bfe71f91112f777919f6a6014e40e6ebbaa25f15a143cb407603\": container with ID starting with 41d9e1e464e8bfe71f91112f777919f6a6014e40e6ebbaa25f15a143cb407603 not found: ID does not exist" containerID="41d9e1e464e8bfe71f91112f777919f6a6014e40e6ebbaa25f15a143cb407603" Sep 08 11:54:53.123911 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.123853 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d9e1e464e8bfe71f91112f777919f6a6014e40e6ebbaa25f15a143cb407603"} err="failed to get container status \"41d9e1e464e8bfe71f91112f777919f6a6014e40e6ebbaa25f15a143cb407603\": rpc error: code = NotFound desc = could not find container \"41d9e1e464e8bfe71f91112f777919f6a6014e40e6ebbaa25f15a143cb407603\": container with ID starting with 41d9e1e464e8bfe71f91112f777919f6a6014e40e6ebbaa25f15a143cb407603 not found: ID does not exist" Sep 08 11:54:53.142212 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.142183 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4"] Sep 08 11:54:53.145422 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.145396 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-77fac0-689d946ffc-648g4"] Sep 08 11:54:53.156585 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.156564 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndr9f\" (UniqueName: \"kubernetes.io/projected/fa47f559-e025-43bd-9867-28b56a844ada-kube-api-access-ndr9f\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:54:53.156585 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.156586 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa47f559-e025-43bd-9867-28b56a844ada-metrics-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:54:53.156711 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.156596 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa47f559-e025-43bd-9867-28b56a844ada-openshift-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:54:53.156711 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.156609 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa47f559-e025-43bd-9867-28b56a844ada-maas-api-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:54:53.654102 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:53.654070 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa47f559-e025-43bd-9867-28b56a844ada" path="/var/lib/kubelet/pods/fa47f559-e025-43bd-9867-28b56a844ada/volumes" Sep 08 11:54:57.438849 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:54:57.438817 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:55:00.555239 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:00.555194 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch"] Sep 08 11:55:00.555753 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:00.555505 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" containerName="istio-proxy" containerID="cri-o://52fcc2b420cc11223c5de1ece1a8b541f48a31c30be18049d5530a3f226ed6fe" gracePeriod=30 Sep 08 11:55:04.889917 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:04.889868 2564 patch_prober.go:28] interesting pod/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 11:55:04.890393 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:04.889942 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 11:55:05.818115 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.818058 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:55:05.891025 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.890989 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f263488-35f0-4110-92b9-460e5ee4c449-istio-podinfo\") pod \"5f263488-35f0-4110-92b9-460e5ee4c449\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " Sep 08 11:55:05.891504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891037 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djsc9\" (UniqueName: \"kubernetes.io/projected/5f263488-35f0-4110-92b9-460e5ee4c449-kube-api-access-djsc9\") pod \"5f263488-35f0-4110-92b9-460e5ee4c449\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " Sep 08 11:55:05.891504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891079 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-workload-certs\") pod \"5f263488-35f0-4110-92b9-460e5ee4c449\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " Sep 08 11:55:05.891504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891147 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f263488-35f0-4110-92b9-460e5ee4c449-istiod-ca-cert\") pod \"5f263488-35f0-4110-92b9-460e5ee4c449\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " Sep 08 11:55:05.891504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891182 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-istio-data\") pod \"5f263488-35f0-4110-92b9-460e5ee4c449\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " Sep 08 11:55:05.891504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891206 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-istio-envoy\") pod \"5f263488-35f0-4110-92b9-460e5ee4c449\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " Sep 08 11:55:05.891504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891247 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-workload-socket\") pod \"5f263488-35f0-4110-92b9-460e5ee4c449\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " Sep 08 11:55:05.891504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891321 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f263488-35f0-4110-92b9-460e5ee4c449-istio-token\") pod \"5f263488-35f0-4110-92b9-460e5ee4c449\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " Sep 08 11:55:05.891504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891417 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-credential-socket\") pod \"5f263488-35f0-4110-92b9-460e5ee4c449\" (UID: \"5f263488-35f0-4110-92b9-460e5ee4c449\") " Sep 08 11:55:05.891504 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891463 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f263488-35f0-4110-92b9-460e5ee4c449" (UID: "5f263488-35f0-4110-92b9-460e5ee4c449"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:55:05.892008 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891665 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f263488-35f0-4110-92b9-460e5ee4c449" (UID: "5f263488-35f0-4110-92b9-460e5ee4c449"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:55:05.892008 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891671 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f263488-35f0-4110-92b9-460e5ee4c449" (UID: "5f263488-35f0-4110-92b9-460e5ee4c449"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:55:05.892008 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891869 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f263488-35f0-4110-92b9-460e5ee4c449" (UID: "5f263488-35f0-4110-92b9-460e5ee4c449"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:55:05.892008 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891880 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-workload-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:05.892008 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891914 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-istio-data\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:05.892008 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891913 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f263488-35f0-4110-92b9-460e5ee4c449-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f263488-35f0-4110-92b9-460e5ee4c449" (UID: "5f263488-35f0-4110-92b9-460e5ee4c449"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:55:05.892008 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.891929 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-workload-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:05.893293 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.893263 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f263488-35f0-4110-92b9-460e5ee4c449-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f263488-35f0-4110-92b9-460e5ee4c449" (UID: "5f263488-35f0-4110-92b9-460e5ee4c449"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 11:55:05.893474 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.893456 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f263488-35f0-4110-92b9-460e5ee4c449-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f263488-35f0-4110-92b9-460e5ee4c449" (UID: "5f263488-35f0-4110-92b9-460e5ee4c449"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:55:05.893534 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.893518 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f263488-35f0-4110-92b9-460e5ee4c449-kube-api-access-djsc9" (OuterVolumeSpecName: "kube-api-access-djsc9") pod "5f263488-35f0-4110-92b9-460e5ee4c449" (UID: "5f263488-35f0-4110-92b9-460e5ee4c449"). InnerVolumeSpecName "kube-api-access-djsc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:55:05.893671 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.893653 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f263488-35f0-4110-92b9-460e5ee4c449" (UID: "5f263488-35f0-4110-92b9-460e5ee4c449"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:55:05.993173 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.993139 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f263488-35f0-4110-92b9-460e5ee4c449-istiod-ca-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:05.993173 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.993172 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-istio-envoy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:05.993408 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.993183 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f263488-35f0-4110-92b9-460e5ee4c449-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:05.993408 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.993195 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f263488-35f0-4110-92b9-460e5ee4c449-credential-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:05.993408 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.993208 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f263488-35f0-4110-92b9-460e5ee4c449-istio-podinfo\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:05.993408 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:05.993222 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djsc9\" (UniqueName: \"kubernetes.io/projected/5f263488-35f0-4110-92b9-460e5ee4c449-kube-api-access-djsc9\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:06.199964 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:06.199929 2564 generic.go:356] "Generic (PLEG): container finished" podID="5f263488-35f0-4110-92b9-460e5ee4c449" containerID="52fcc2b420cc11223c5de1ece1a8b541f48a31c30be18049d5530a3f226ed6fe" exitCode=0 Sep 08 11:55:06.200134 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:06.199976 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" Sep 08 11:55:06.200134 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:06.199980 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" event={"ID":"5f263488-35f0-4110-92b9-460e5ee4c449","Type":"ContainerDied","Data":"52fcc2b420cc11223c5de1ece1a8b541f48a31c30be18049d5530a3f226ed6fe"} Sep 08 11:55:06.200134 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:06.200024 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch" event={"ID":"5f263488-35f0-4110-92b9-460e5ee4c449","Type":"ContainerDied","Data":"e2a74358cce06106aad3a80ab9804d7152c38e61b4b98b9e1b741e4f91014590"} Sep 08 11:55:06.200134 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:06.200045 2564 scope.go:117] "RemoveContainer" containerID="52fcc2b420cc11223c5de1ece1a8b541f48a31c30be18049d5530a3f226ed6fe" Sep 08 11:55:06.210566 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:06.210522 2564 scope.go:117] "RemoveContainer" containerID="ec53031833862c545be600c8ac049b10ce5008258c112fbeab52a871def12346" Sep 08 11:55:06.218614 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:06.218595 2564 scope.go:117] "RemoveContainer" containerID="52fcc2b420cc11223c5de1ece1a8b541f48a31c30be18049d5530a3f226ed6fe" Sep 08 11:55:06.218866 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:55:06.218847 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52fcc2b420cc11223c5de1ece1a8b541f48a31c30be18049d5530a3f226ed6fe\": container with ID starting with 52fcc2b420cc11223c5de1ece1a8b541f48a31c30be18049d5530a3f226ed6fe not found: ID does not exist" containerID="52fcc2b420cc11223c5de1ece1a8b541f48a31c30be18049d5530a3f226ed6fe" Sep 08 11:55:06.218944 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:06.218878 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52fcc2b420cc11223c5de1ece1a8b541f48a31c30be18049d5530a3f226ed6fe"} err="failed to get container status \"52fcc2b420cc11223c5de1ece1a8b541f48a31c30be18049d5530a3f226ed6fe\": rpc error: code = NotFound desc = could not find container \"52fcc2b420cc11223c5de1ece1a8b541f48a31c30be18049d5530a3f226ed6fe\": container with ID starting with 52fcc2b420cc11223c5de1ece1a8b541f48a31c30be18049d5530a3f226ed6fe not found: ID does not exist" Sep 08 11:55:06.218944 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:06.218904 2564 scope.go:117] "RemoveContainer" containerID="ec53031833862c545be600c8ac049b10ce5008258c112fbeab52a871def12346" Sep 08 11:55:06.219172 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:55:06.219155 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec53031833862c545be600c8ac049b10ce5008258c112fbeab52a871def12346\": container with ID starting with ec53031833862c545be600c8ac049b10ce5008258c112fbeab52a871def12346 not found: ID does not exist" containerID="ec53031833862c545be600c8ac049b10ce5008258c112fbeab52a871def12346" Sep 08 11:55:06.219217 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:06.219178 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec53031833862c545be600c8ac049b10ce5008258c112fbeab52a871def12346"} err="failed to get container status \"ec53031833862c545be600c8ac049b10ce5008258c112fbeab52a871def12346\": rpc error: code = NotFound desc = could not find container \"ec53031833862c545be600c8ac049b10ce5008258c112fbeab52a871def12346\": container with ID starting with ec53031833862c545be600c8ac049b10ce5008258c112fbeab52a871def12346 not found: ID does not exist" Sep 08 11:55:06.225497 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:06.225475 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch"] Sep 08 11:55:06.230536 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:06.230515 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-77fac0-openshift-default-6ccc45cd5b-q7rch"] Sep 08 11:55:07.653779 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:07.653746 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" path="/var/lib/kubelet/pods/5f263488-35f0-4110-92b9-460e5ee4c449/volumes" Sep 08 11:55:11.716781 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:11.716751 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 11:55:11.718881 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:11.718858 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 11:55:14.433057 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:14.433017 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:55:16.448109 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:16.448077 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb"] Sep 08 11:55:16.448568 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:16.448281 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" podUID="947f2822-1b50-432f-9d97-9e077cf0d554" containerName="maas-api" containerID="cri-o://72b3d6ccde4f71102edc8e74a214d7c5cebc69136f319dc9805be2887dc77c44" gracePeriod=30 Sep 08 11:55:16.702696 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:16.702636 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:55:16.805500 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:16.805468 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/947f2822-1b50-432f-9d97-9e077cf0d554-openshift-service-ca\") pod \"947f2822-1b50-432f-9d97-9e077cf0d554\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " Sep 08 11:55:16.805651 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:16.805525 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/947f2822-1b50-432f-9d97-9e077cf0d554-metrics-certs\") pod \"947f2822-1b50-432f-9d97-9e077cf0d554\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " Sep 08 11:55:16.805651 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:16.805550 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndrdp\" (UniqueName: \"kubernetes.io/projected/947f2822-1b50-432f-9d97-9e077cf0d554-kube-api-access-ndrdp\") pod \"947f2822-1b50-432f-9d97-9e077cf0d554\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " Sep 08 11:55:16.805651 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:16.805583 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/947f2822-1b50-432f-9d97-9e077cf0d554-maas-api-tls\") pod \"947f2822-1b50-432f-9d97-9e077cf0d554\" (UID: \"947f2822-1b50-432f-9d97-9e077cf0d554\") " Sep 08 11:55:16.805817 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:16.805798 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947f2822-1b50-432f-9d97-9e077cf0d554-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "947f2822-1b50-432f-9d97-9e077cf0d554" (UID: "947f2822-1b50-432f-9d97-9e077cf0d554"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:55:16.805981 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:16.805962 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/947f2822-1b50-432f-9d97-9e077cf0d554-openshift-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:16.807634 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:16.807610 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947f2822-1b50-432f-9d97-9e077cf0d554-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "947f2822-1b50-432f-9d97-9e077cf0d554" (UID: "947f2822-1b50-432f-9d97-9e077cf0d554"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:55:16.807740 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:16.807628 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947f2822-1b50-432f-9d97-9e077cf0d554-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "947f2822-1b50-432f-9d97-9e077cf0d554" (UID: "947f2822-1b50-432f-9d97-9e077cf0d554"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:55:16.807740 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:16.807665 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947f2822-1b50-432f-9d97-9e077cf0d554-kube-api-access-ndrdp" (OuterVolumeSpecName: "kube-api-access-ndrdp") pod "947f2822-1b50-432f-9d97-9e077cf0d554" (UID: "947f2822-1b50-432f-9d97-9e077cf0d554"). InnerVolumeSpecName "kube-api-access-ndrdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:55:16.906718 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:16.906685 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/947f2822-1b50-432f-9d97-9e077cf0d554-metrics-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:16.906718 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:16.906712 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndrdp\" (UniqueName: \"kubernetes.io/projected/947f2822-1b50-432f-9d97-9e077cf0d554-kube-api-access-ndrdp\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:16.906718 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:16.906721 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/947f2822-1b50-432f-9d97-9e077cf0d554-maas-api-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:17.277284 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:17.277249 2564 generic.go:356] "Generic (PLEG): container finished" podID="947f2822-1b50-432f-9d97-9e077cf0d554" containerID="72b3d6ccde4f71102edc8e74a214d7c5cebc69136f319dc9805be2887dc77c44" exitCode=0 Sep 08 11:55:17.277501 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:17.277318 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" Sep 08 11:55:17.277501 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:17.277344 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" event={"ID":"947f2822-1b50-432f-9d97-9e077cf0d554","Type":"ContainerDied","Data":"72b3d6ccde4f71102edc8e74a214d7c5cebc69136f319dc9805be2887dc77c44"} Sep 08 11:55:17.277501 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:17.277429 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb" event={"ID":"947f2822-1b50-432f-9d97-9e077cf0d554","Type":"ContainerDied","Data":"e978752380c52968fed3fb3570f755e4e45bc7aa2f56e0054bf23e5125522b94"} Sep 08 11:55:17.277501 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:17.277461 2564 scope.go:117] "RemoveContainer" containerID="72b3d6ccde4f71102edc8e74a214d7c5cebc69136f319dc9805be2887dc77c44" Sep 08 11:55:17.288187 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:17.288172 2564 scope.go:117] "RemoveContainer" containerID="72b3d6ccde4f71102edc8e74a214d7c5cebc69136f319dc9805be2887dc77c44" Sep 08 11:55:17.288508 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:55:17.288488 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b3d6ccde4f71102edc8e74a214d7c5cebc69136f319dc9805be2887dc77c44\": container with ID starting with 72b3d6ccde4f71102edc8e74a214d7c5cebc69136f319dc9805be2887dc77c44 not found: ID does not exist" containerID="72b3d6ccde4f71102edc8e74a214d7c5cebc69136f319dc9805be2887dc77c44" Sep 08 11:55:17.288569 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:17.288515 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b3d6ccde4f71102edc8e74a214d7c5cebc69136f319dc9805be2887dc77c44"} err="failed to get container status \"72b3d6ccde4f71102edc8e74a214d7c5cebc69136f319dc9805be2887dc77c44\": rpc error: code = NotFound desc = could not find container \"72b3d6ccde4f71102edc8e74a214d7c5cebc69136f319dc9805be2887dc77c44\": container with ID starting with 72b3d6ccde4f71102edc8e74a214d7c5cebc69136f319dc9805be2887dc77c44 not found: ID does not exist" Sep 08 11:55:17.304027 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:17.303998 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb"] Sep 08 11:55:17.308899 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:17.308876 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ca8280-ddbb69ff8-dztpb"] Sep 08 11:55:17.653918 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:17.653879 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="947f2822-1b50-432f-9d97-9e077cf0d554" path="/var/lib/kubelet/pods/947f2822-1b50-432f-9d97-9e077cf0d554/volumes" Sep 08 11:55:23.834773 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:23.834733 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr"] Sep 08 11:55:23.835260 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:23.834985 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" podUID="bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" containerName="istio-proxy" containerID="cri-o://d2b17197429769eba2725346818c9b3f943986503f25d1e9a48741c1996cc3f6" gracePeriod=30 Sep 08 11:55:29.090066 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.090041 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:55:29.124607 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.124542 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-podinfo\") pod \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " Sep 08 11:55:29.124607 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.124581 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-workload-socket\") pod \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " Sep 08 11:55:29.124819 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.124634 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istiod-ca-cert\") pod \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " Sep 08 11:55:29.124819 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.124666 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vpmv\" (UniqueName: \"kubernetes.io/projected/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-kube-api-access-6vpmv\") pod \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " Sep 08 11:55:29.124819 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.124732 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-envoy\") pod \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " Sep 08 11:55:29.124819 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.124762 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-data\") pod \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " Sep 08 11:55:29.125027 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.124826 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-workload-certs\") pod \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " Sep 08 11:55:29.125027 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.124856 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-credential-socket\") pod \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " Sep 08 11:55:29.125027 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.124883 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-token\") pod \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\" (UID: \"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a\") " Sep 08 11:55:29.125027 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.124880 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" (UID: "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:55:29.125235 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.125054 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" (UID: "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:55:29.125235 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.125158 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" (UID: "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:55:29.125553 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.125351 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-workload-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:29.125553 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.125422 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istiod-ca-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:29.125553 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.125439 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-workload-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:29.125553 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.125390 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" (UID: "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:55:29.125842 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.125584 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-data" (OuterVolumeSpecName: "istio-data") pod "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" (UID: "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:55:29.127226 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.127199 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" (UID: "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 11:55:29.127334 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.127314 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-kube-api-access-6vpmv" (OuterVolumeSpecName: "kube-api-access-6vpmv") pod "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" (UID: "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a"). InnerVolumeSpecName "kube-api-access-6vpmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:55:29.127413 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.127342 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" (UID: "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:55:29.127574 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.127545 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-token" (OuterVolumeSpecName: "istio-token") pod "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" (UID: "bb81f47f-e110-4d24-a80d-cc2ada6a5e3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:55:29.225960 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.225931 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vpmv\" (UniqueName: \"kubernetes.io/projected/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-kube-api-access-6vpmv\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:29.225960 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.225954 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-envoy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:29.225960 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.225963 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-data\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:29.226177 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.225972 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-credential-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:29.226177 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.225981 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:29.226177 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.225989 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a-istio-podinfo\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:29.362152 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.362119 2564 generic.go:356] "Generic (PLEG): container finished" podID="bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" containerID="d2b17197429769eba2725346818c9b3f943986503f25d1e9a48741c1996cc3f6" exitCode=0 Sep 08 11:55:29.362317 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.362174 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" Sep 08 11:55:29.362317 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.362197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" event={"ID":"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a","Type":"ContainerDied","Data":"d2b17197429769eba2725346818c9b3f943986503f25d1e9a48741c1996cc3f6"} Sep 08 11:55:29.362317 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.362253 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr" event={"ID":"bb81f47f-e110-4d24-a80d-cc2ada6a5e3a","Type":"ContainerDied","Data":"861c35341175d827a5637bc667aa49da515e4a2ac9fc2588438ef03a68524c61"} Sep 08 11:55:29.362317 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.362275 2564 scope.go:117] "RemoveContainer" containerID="d2b17197429769eba2725346818c9b3f943986503f25d1e9a48741c1996cc3f6" Sep 08 11:55:29.372361 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.372346 2564 scope.go:117] "RemoveContainer" containerID="5f9f9cff6dffcb9ce108b067c0838235950097e0d1f021175d9f8b6f7db47090" Sep 08 11:55:29.381576 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.381557 2564 scope.go:117] "RemoveContainer" containerID="d2b17197429769eba2725346818c9b3f943986503f25d1e9a48741c1996cc3f6" Sep 08 11:55:29.381849 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:55:29.381830 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b17197429769eba2725346818c9b3f943986503f25d1e9a48741c1996cc3f6\": container with ID starting with d2b17197429769eba2725346818c9b3f943986503f25d1e9a48741c1996cc3f6 not found: ID does not exist" containerID="d2b17197429769eba2725346818c9b3f943986503f25d1e9a48741c1996cc3f6" Sep 08 11:55:29.381895 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.381858 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b17197429769eba2725346818c9b3f943986503f25d1e9a48741c1996cc3f6"} err="failed to get container status \"d2b17197429769eba2725346818c9b3f943986503f25d1e9a48741c1996cc3f6\": rpc error: code = NotFound desc = could not find container \"d2b17197429769eba2725346818c9b3f943986503f25d1e9a48741c1996cc3f6\": container with ID starting with d2b17197429769eba2725346818c9b3f943986503f25d1e9a48741c1996cc3f6 not found: ID does not exist" Sep 08 11:55:29.381895 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.381875 2564 scope.go:117] "RemoveContainer" containerID="5f9f9cff6dffcb9ce108b067c0838235950097e0d1f021175d9f8b6f7db47090" Sep 08 11:55:29.382132 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:55:29.382113 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9f9cff6dffcb9ce108b067c0838235950097e0d1f021175d9f8b6f7db47090\": container with ID starting with 5f9f9cff6dffcb9ce108b067c0838235950097e0d1f021175d9f8b6f7db47090 not found: ID does not exist" containerID="5f9f9cff6dffcb9ce108b067c0838235950097e0d1f021175d9f8b6f7db47090" Sep 08 11:55:29.382193 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.382143 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9f9cff6dffcb9ce108b067c0838235950097e0d1f021175d9f8b6f7db47090"} err="failed to get container status \"5f9f9cff6dffcb9ce108b067c0838235950097e0d1f021175d9f8b6f7db47090\": rpc error: code = NotFound desc = could not find container \"5f9f9cff6dffcb9ce108b067c0838235950097e0d1f021175d9f8b6f7db47090\": container with ID starting with 5f9f9cff6dffcb9ce108b067c0838235950097e0d1f021175d9f8b6f7db47090 not found: ID does not exist" Sep 08 11:55:29.388196 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.388173 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr"] Sep 08 11:55:29.399039 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.399010 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ca8280-openshift-default-5b67d7d6db-47ppr"] Sep 08 11:55:29.654082 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:29.654049 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" path="/var/lib/kubelet/pods/bb81f47f-e110-4d24-a80d-cc2ada6a5e3a/volumes" Sep 08 11:55:31.340678 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:31.340633 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:55:42.007064 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:42.006968 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc"] Sep 08 11:55:42.007614 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:42.007214 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" podUID="6b3c3e85-998c-4d66-b921-0c97dab30697" containerName="istio-proxy" containerID="cri-o://5f76f755fe6f583585bd666d25386ed225fd53464abb91f0555f6952c35539b8" gracePeriod=30 Sep 08 11:55:47.108229 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108196 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv"] Sep 08 11:55:47.108662 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108646 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" containerName="istio-proxy" Sep 08 11:55:47.108714 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108664 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" containerName="istio-proxy" Sep 08 11:55:47.108714 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108678 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa47f559-e025-43bd-9867-28b56a844ada" containerName="maas-api" Sep 08 11:55:47.108714 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108684 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa47f559-e025-43bd-9867-28b56a844ada" containerName="maas-api" Sep 08 11:55:47.108714 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108691 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" containerName="istio-proxy" Sep 08 11:55:47.108714 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108696 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" containerName="istio-proxy" Sep 08 11:55:47.108714 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108703 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" containerName="istio-proxy" Sep 08 11:55:47.108714 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108708 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" containerName="istio-proxy" Sep 08 11:55:47.108952 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108718 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" containerName="istio-proxy" Sep 08 11:55:47.108952 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108723 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" containerName="istio-proxy" Sep 08 11:55:47.108952 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108735 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="947f2822-1b50-432f-9d97-9e077cf0d554" containerName="maas-api" Sep 08 11:55:47.108952 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108740 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="947f2822-1b50-432f-9d97-9e077cf0d554" containerName="maas-api" Sep 08 11:55:47.108952 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108817 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="947f2822-1b50-432f-9d97-9e077cf0d554" containerName="maas-api" Sep 08 11:55:47.108952 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108825 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa47f559-e025-43bd-9867-28b56a844ada" containerName="maas-api" Sep 08 11:55:47.108952 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108834 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" containerName="istio-proxy" Sep 08 11:55:47.108952 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108841 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f263488-35f0-4110-92b9-460e5ee4c449" containerName="istio-proxy" Sep 08 11:55:47.108952 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108846 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" containerName="istio-proxy" Sep 08 11:55:47.108952 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.108854 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb81f47f-e110-4d24-a80d-cc2ada6a5e3a" containerName="istio-proxy" Sep 08 11:55:47.112088 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.112066 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.115282 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.115248 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-64178b-openshift-default-dockercfg-qxxns\"" Sep 08 11:55:47.126703 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.126678 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv"] Sep 08 11:55:47.199392 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.199243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-podinfo\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.199392 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.199331 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6zcn\" (UniqueName: \"kubernetes.io/projected/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-kube-api-access-s6zcn\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.199634 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.199399 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-envoy\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.199634 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.199435 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-workload-socket\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.199634 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.199469 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istiod-ca-cert\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.199634 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.199562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-data\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.199853 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.199651 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-credential-socket\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.199853 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.199673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-workload-certs\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.199853 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.199716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-token\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.286979 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.286912 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:55:47.300814 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.300780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-data\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.300956 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.300854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-credential-socket\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.300956 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.300879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-workload-certs\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.300956 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.300929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-token\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.301126 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.300969 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-podinfo\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.301126 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.301005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6zcn\" (UniqueName: \"kubernetes.io/projected/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-kube-api-access-s6zcn\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.301126 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.301032 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-envoy\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.301126 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.301067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-workload-socket\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.301126 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.301100 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istiod-ca-cert\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.301409 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.301173 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-data\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.301409 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.301265 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-workload-certs\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.301532 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.301452 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-workload-socket\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.301595 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.301551 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-credential-socket\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.301798 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.301776 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istiod-ca-cert\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.303320 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.303294 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-envoy\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.303446 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.303416 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-podinfo\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.314118 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.314079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-token\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.315672 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.315651 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6zcn\" (UniqueName: \"kubernetes.io/projected/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-kube-api-access-s6zcn\") pod \"e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.401706 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.401665 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-podinfo\") pod \"6b3c3e85-998c-4d66-b921-0c97dab30697\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " Sep 08 11:55:47.401706 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.401712 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-workload-certs\") pod \"6b3c3e85-998c-4d66-b921-0c97dab30697\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " Sep 08 11:55:47.401939 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.401819 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-data\") pod \"6b3c3e85-998c-4d66-b921-0c97dab30697\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " Sep 08 11:55:47.401939 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.401860 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-envoy\") pod \"6b3c3e85-998c-4d66-b921-0c97dab30697\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " Sep 08 11:55:47.401939 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.401896 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b3c3e85-998c-4d66-b921-0c97dab30697-istiod-ca-cert\") pod \"6b3c3e85-998c-4d66-b921-0c97dab30697\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " Sep 08 11:55:47.401939 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.401925 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkpxm\" (UniqueName: \"kubernetes.io/projected/6b3c3e85-998c-4d66-b921-0c97dab30697-kube-api-access-mkpxm\") pod \"6b3c3e85-998c-4d66-b921-0c97dab30697\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " Sep 08 11:55:47.402162 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.401975 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-workload-socket\") pod \"6b3c3e85-998c-4d66-b921-0c97dab30697\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " Sep 08 11:55:47.402162 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.402038 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b3c3e85-998c-4d66-b921-0c97dab30697" (UID: "6b3c3e85-998c-4d66-b921-0c97dab30697"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:55:47.402162 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.402087 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b3c3e85-998c-4d66-b921-0c97dab30697" (UID: "6b3c3e85-998c-4d66-b921-0c97dab30697"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:55:47.402162 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.402101 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-token\") pod \"6b3c3e85-998c-4d66-b921-0c97dab30697\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " Sep 08 11:55:47.402351 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.402199 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-credential-socket\") pod \"6b3c3e85-998c-4d66-b921-0c97dab30697\" (UID: \"6b3c3e85-998c-4d66-b921-0c97dab30697\") " Sep 08 11:55:47.402351 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.402283 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b3c3e85-998c-4d66-b921-0c97dab30697" (UID: "6b3c3e85-998c-4d66-b921-0c97dab30697"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:55:47.402474 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.402433 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3c3e85-998c-4d66-b921-0c97dab30697-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b3c3e85-998c-4d66-b921-0c97dab30697" (UID: "6b3c3e85-998c-4d66-b921-0c97dab30697"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:55:47.402605 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.402581 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b3c3e85-998c-4d66-b921-0c97dab30697" (UID: "6b3c3e85-998c-4d66-b921-0c97dab30697"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:55:47.402769 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.402751 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-workload-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:47.402842 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.402773 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-data\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:47.402842 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.402788 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b3c3e85-998c-4d66-b921-0c97dab30697-istiod-ca-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:47.402842 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.402800 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-workload-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:47.402842 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.402815 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-credential-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:47.403957 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.403928 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b3c3e85-998c-4d66-b921-0c97dab30697" (UID: "6b3c3e85-998c-4d66-b921-0c97dab30697"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 11:55:47.404156 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.404134 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3c3e85-998c-4d66-b921-0c97dab30697-kube-api-access-mkpxm" (OuterVolumeSpecName: "kube-api-access-mkpxm") pod "6b3c3e85-998c-4d66-b921-0c97dab30697" (UID: "6b3c3e85-998c-4d66-b921-0c97dab30697"). InnerVolumeSpecName "kube-api-access-mkpxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:55:47.404217 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.404142 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b3c3e85-998c-4d66-b921-0c97dab30697" (UID: "6b3c3e85-998c-4d66-b921-0c97dab30697"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:55:47.404255 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.404207 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b3c3e85-998c-4d66-b921-0c97dab30697" (UID: "6b3c3e85-998c-4d66-b921-0c97dab30697"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:55:47.423549 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.423526 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:47.473059 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.473025 2564 generic.go:356] "Generic (PLEG): container finished" podID="6b3c3e85-998c-4d66-b921-0c97dab30697" containerID="5f76f755fe6f583585bd666d25386ed225fd53464abb91f0555f6952c35539b8" exitCode=0 Sep 08 11:55:47.473360 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.473105 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" Sep 08 11:55:47.473360 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.473120 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" event={"ID":"6b3c3e85-998c-4d66-b921-0c97dab30697","Type":"ContainerDied","Data":"5f76f755fe6f583585bd666d25386ed225fd53464abb91f0555f6952c35539b8"} Sep 08 11:55:47.473360 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.473222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc" event={"ID":"6b3c3e85-998c-4d66-b921-0c97dab30697","Type":"ContainerDied","Data":"c8852a278aa8f89ea73e7fc80ca6bcc0a6b614b8b8f3f76cf06f4456d228c578"} Sep 08 11:55:47.473360 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.473255 2564 scope.go:117] "RemoveContainer" containerID="5f76f755fe6f583585bd666d25386ed225fd53464abb91f0555f6952c35539b8" Sep 08 11:55:47.495923 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.495903 2564 scope.go:117] "RemoveContainer" containerID="53c3f61dd8e280dc52e5525371030498bc48a6d3c909a4a6439a49e9708521cc" Sep 08 11:55:47.504226 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.504200 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-envoy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:47.504352 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.504231 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkpxm\" (UniqueName: \"kubernetes.io/projected/6b3c3e85-998c-4d66-b921-0c97dab30697-kube-api-access-mkpxm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:47.504352 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.504247 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:47.504352 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.504261 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b3c3e85-998c-4d66-b921-0c97dab30697-istio-podinfo\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:47.505770 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.505750 2564 scope.go:117] "RemoveContainer" containerID="5f76f755fe6f583585bd666d25386ed225fd53464abb91f0555f6952c35539b8" Sep 08 11:55:47.506108 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:55:47.506084 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f76f755fe6f583585bd666d25386ed225fd53464abb91f0555f6952c35539b8\": container with ID starting with 5f76f755fe6f583585bd666d25386ed225fd53464abb91f0555f6952c35539b8 not found: ID does not exist" containerID="5f76f755fe6f583585bd666d25386ed225fd53464abb91f0555f6952c35539b8" Sep 08 11:55:47.506198 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.506115 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f76f755fe6f583585bd666d25386ed225fd53464abb91f0555f6952c35539b8"} err="failed to get container status \"5f76f755fe6f583585bd666d25386ed225fd53464abb91f0555f6952c35539b8\": rpc error: code = NotFound desc = could not find container \"5f76f755fe6f583585bd666d25386ed225fd53464abb91f0555f6952c35539b8\": container with ID starting with 5f76f755fe6f583585bd666d25386ed225fd53464abb91f0555f6952c35539b8 not found: ID does not exist" Sep 08 11:55:47.506198 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.506134 2564 scope.go:117] "RemoveContainer" containerID="53c3f61dd8e280dc52e5525371030498bc48a6d3c909a4a6439a49e9708521cc" Sep 08 11:55:47.506456 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:55:47.506440 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c3f61dd8e280dc52e5525371030498bc48a6d3c909a4a6439a49e9708521cc\": container with ID starting with 53c3f61dd8e280dc52e5525371030498bc48a6d3c909a4a6439a49e9708521cc not found: ID does not exist" containerID="53c3f61dd8e280dc52e5525371030498bc48a6d3c909a4a6439a49e9708521cc" Sep 08 11:55:47.506516 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.506463 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c3f61dd8e280dc52e5525371030498bc48a6d3c909a4a6439a49e9708521cc"} err="failed to get container status \"53c3f61dd8e280dc52e5525371030498bc48a6d3c909a4a6439a49e9708521cc\": rpc error: code = NotFound desc = could not find container \"53c3f61dd8e280dc52e5525371030498bc48a6d3c909a4a6439a49e9708521cc\": container with ID starting with 53c3f61dd8e280dc52e5525371030498bc48a6d3c909a4a6439a49e9708521cc not found: ID does not exist" Sep 08 11:55:47.506957 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.506939 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc"] Sep 08 11:55:47.512889 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.512866 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-69606d-openshift-default-6cbc56f7c6-qc7fc"] Sep 08 11:55:47.653646 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.653613 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b3c3e85-998c-4d66-b921-0c97dab30697" path="/var/lib/kubelet/pods/6b3c3e85-998c-4d66-b921-0c97dab30697/volumes" Sep 08 11:55:47.710927 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.710898 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv"] Sep 08 11:55:47.711642 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:55:47.711615 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8fb99c_cdee_4ed4_b371_d7ebd1de73ba.slice/crio-d6d2a97aef8f936baceb810798e56c3e8dd7b5c8ec0953a2ad2a5c2c1fd96a57 WatchSource:0}: Error finding container d6d2a97aef8f936baceb810798e56c3e8dd7b5c8ec0953a2ad2a5c2c1fd96a57: Status 404 returned error can't find the container with id d6d2a97aef8f936baceb810798e56c3e8dd7b5c8ec0953a2ad2a5c2c1fd96a57 Sep 08 11:55:47.713715 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.713687 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:55:47.713779 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.713748 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:55:47.713822 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:47.713782 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:55:48.245323 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:48.245288 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:55:48.483310 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:48.483277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" event={"ID":"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba","Type":"ContainerStarted","Data":"a27ef6a61230cc204756bd3ec929c9750620b9a2b82670fd1fa59c081c6c93f7"} Sep 08 11:55:48.483521 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:48.483328 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" event={"ID":"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba","Type":"ContainerStarted","Data":"d6d2a97aef8f936baceb810798e56c3e8dd7b5c8ec0953a2ad2a5c2c1fd96a57"} Sep 08 11:55:48.508074 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:48.508023 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" podStartSLOduration=1.5080061790000001 podStartE2EDuration="1.508006179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:55:48.504985306 +0000 UTC m=+1537.364172643" watchObservedRunningTime="2026-09-08 11:55:48.508006179 +0000 UTC m=+1537.367193496" Sep 08 11:55:49.423690 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:49.423653 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:49.429347 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:49.429322 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:49.489718 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:49.489687 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:49.490700 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:49.490680 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:55:51.517101 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.517066 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp"] Sep 08 11:55:51.517603 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.517586 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3c3e85-998c-4d66-b921-0c97dab30697" containerName="istio-proxy" Sep 08 11:55:51.517676 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.517605 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3c3e85-998c-4d66-b921-0c97dab30697" containerName="istio-proxy" Sep 08 11:55:51.517676 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.517616 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3c3e85-998c-4d66-b921-0c97dab30697" containerName="istio-proxy" Sep 08 11:55:51.517676 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.517623 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3c3e85-998c-4d66-b921-0c97dab30697" containerName="istio-proxy" Sep 08 11:55:51.517800 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.517702 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b3c3e85-998c-4d66-b921-0c97dab30697" containerName="istio-proxy" Sep 08 11:55:51.517800 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.517712 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b3c3e85-998c-4d66-b921-0c97dab30697" containerName="istio-proxy" Sep 08 11:55:51.521157 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.521131 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:55:51.523968 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.523945 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-64178b-serving-cert\"" Sep 08 11:55:51.533633 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.533602 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp"] Sep 08 11:55:51.645822 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.645788 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxhgq\" (UniqueName: \"kubernetes.io/projected/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-kube-api-access-mxhgq\") pod \"maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:55:51.646025 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.645850 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:55:51.646025 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.646001 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:55:51.646153 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.646064 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:55:51.746991 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.746953 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:55:51.747239 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.747010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:55:51.747239 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.747041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxhgq\" (UniqueName: \"kubernetes.io/projected/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-kube-api-access-mxhgq\") pod \"maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:55:51.747239 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.747083 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:55:51.747746 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.747726 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:55:51.749572 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.749538 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:55:51.749675 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.749614 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:55:51.758029 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.758005 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxhgq\" (UniqueName: \"kubernetes.io/projected/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-kube-api-access-mxhgq\") pod \"maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:55:51.835446 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:51.835331 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:55:52.123001 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.122823 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp"] Sep 08 11:55:52.125319 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:55:52.125294 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829bbc4f_fc2d_4db0_9354_ca6acc2c05fe.slice/crio-6c38fcb4c4ce2ab5ba174792b6d0e18fe949d2194501a070623e1ba2122bfe8b WatchSource:0}: Error finding container 6c38fcb4c4ce2ab5ba174792b6d0e18fe949d2194501a070623e1ba2122bfe8b: Status 404 returned error can't find the container with id 6c38fcb4c4ce2ab5ba174792b6d0e18fe949d2194501a070623e1ba2122bfe8b Sep 08 11:55:52.445489 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.445424 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk"] Sep 08 11:55:52.446332 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.445728 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" podUID="5a0ea850-5f36-482b-9719-58c6f16a395e" containerName="maas-api" containerID="cri-o://70a7bed115cb1c5a5f3da06d3056b5c61c5dbf690ec8495663b3d3cf0dca8d0c" gracePeriod=30 Sep 08 11:55:52.510584 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.510552 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" event={"ID":"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe","Type":"ContainerStarted","Data":"6c38fcb4c4ce2ab5ba174792b6d0e18fe949d2194501a070623e1ba2122bfe8b"} Sep 08 11:55:52.693587 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.693564 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:55:52.759410 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.759363 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5khvx\" (UniqueName: \"kubernetes.io/projected/5a0ea850-5f36-482b-9719-58c6f16a395e-kube-api-access-5khvx\") pod \"5a0ea850-5f36-482b-9719-58c6f16a395e\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " Sep 08 11:55:52.759546 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.759461 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0ea850-5f36-482b-9719-58c6f16a395e-maas-api-tls\") pod \"5a0ea850-5f36-482b-9719-58c6f16a395e\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " Sep 08 11:55:52.759927 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.759555 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0ea850-5f36-482b-9719-58c6f16a395e-metrics-certs\") pod \"5a0ea850-5f36-482b-9719-58c6f16a395e\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " Sep 08 11:55:52.759927 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.759602 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0ea850-5f36-482b-9719-58c6f16a395e-openshift-service-ca\") pod \"5a0ea850-5f36-482b-9719-58c6f16a395e\" (UID: \"5a0ea850-5f36-482b-9719-58c6f16a395e\") " Sep 08 11:55:52.760674 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.760378 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0ea850-5f36-482b-9719-58c6f16a395e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a0ea850-5f36-482b-9719-58c6f16a395e" (UID: "5a0ea850-5f36-482b-9719-58c6f16a395e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:55:52.761186 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.761163 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0ea850-5f36-482b-9719-58c6f16a395e-openshift-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:52.761966 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.761936 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0ea850-5f36-482b-9719-58c6f16a395e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5a0ea850-5f36-482b-9719-58c6f16a395e" (UID: "5a0ea850-5f36-482b-9719-58c6f16a395e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:55:52.762052 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.761966 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0ea850-5f36-482b-9719-58c6f16a395e-kube-api-access-5khvx" (OuterVolumeSpecName: "kube-api-access-5khvx") pod "5a0ea850-5f36-482b-9719-58c6f16a395e" (UID: "5a0ea850-5f36-482b-9719-58c6f16a395e"). InnerVolumeSpecName "kube-api-access-5khvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:55:52.762052 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.761970 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0ea850-5f36-482b-9719-58c6f16a395e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a0ea850-5f36-482b-9719-58c6f16a395e" (UID: "5a0ea850-5f36-482b-9719-58c6f16a395e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:55:52.862793 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.862759 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5khvx\" (UniqueName: \"kubernetes.io/projected/5a0ea850-5f36-482b-9719-58c6f16a395e-kube-api-access-5khvx\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:52.862793 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.862793 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0ea850-5f36-482b-9719-58c6f16a395e-maas-api-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:52.863002 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:52.862806 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0ea850-5f36-482b-9719-58c6f16a395e-metrics-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:55:53.517945 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:53.517893 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" event={"ID":"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe","Type":"ContainerStarted","Data":"5b137e4677d6187af5b20c54215a19689b60de8e4f6f7210513b69a2bd3b2feb"} Sep 08 11:55:53.518153 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:53.517974 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:55:53.519189 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:53.519160 2564 generic.go:356] "Generic (PLEG): container finished" podID="5a0ea850-5f36-482b-9719-58c6f16a395e" containerID="70a7bed115cb1c5a5f3da06d3056b5c61c5dbf690ec8495663b3d3cf0dca8d0c" exitCode=0 Sep 08 11:55:53.519345 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:53.519200 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" Sep 08 11:55:53.519345 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:53.519240 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" event={"ID":"5a0ea850-5f36-482b-9719-58c6f16a395e","Type":"ContainerDied","Data":"70a7bed115cb1c5a5f3da06d3056b5c61c5dbf690ec8495663b3d3cf0dca8d0c"} Sep 08 11:55:53.519345 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:53.519285 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk" event={"ID":"5a0ea850-5f36-482b-9719-58c6f16a395e","Type":"ContainerDied","Data":"f669aadf910ed4d3cc6471692fdba083177cc915bfe35ca844aad556ae40549d"} Sep 08 11:55:53.519345 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:53.519308 2564 scope.go:117] "RemoveContainer" containerID="70a7bed115cb1c5a5f3da06d3056b5c61c5dbf690ec8495663b3d3cf0dca8d0c" Sep 08 11:55:53.529020 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:53.528996 2564 scope.go:117] "RemoveContainer" containerID="70a7bed115cb1c5a5f3da06d3056b5c61c5dbf690ec8495663b3d3cf0dca8d0c" Sep 08 11:55:53.529279 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:55:53.529258 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a7bed115cb1c5a5f3da06d3056b5c61c5dbf690ec8495663b3d3cf0dca8d0c\": container with ID starting with 70a7bed115cb1c5a5f3da06d3056b5c61c5dbf690ec8495663b3d3cf0dca8d0c not found: ID does not exist" containerID="70a7bed115cb1c5a5f3da06d3056b5c61c5dbf690ec8495663b3d3cf0dca8d0c" Sep 08 11:55:53.529347 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:53.529290 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a7bed115cb1c5a5f3da06d3056b5c61c5dbf690ec8495663b3d3cf0dca8d0c"} err="failed to get container status \"70a7bed115cb1c5a5f3da06d3056b5c61c5dbf690ec8495663b3d3cf0dca8d0c\": rpc error: code = NotFound desc = could not find container \"70a7bed115cb1c5a5f3da06d3056b5c61c5dbf690ec8495663b3d3cf0dca8d0c\": container with ID starting with 70a7bed115cb1c5a5f3da06d3056b5c61c5dbf690ec8495663b3d3cf0dca8d0c not found: ID does not exist" Sep 08 11:55:53.536572 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:53.536531 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" podStartSLOduration=2.164550749 podStartE2EDuration="2.53651864s" totalImagesPullingTime="371.967891ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 11:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 11:55:53.536075721 +0000 UTC m=+1542.395263063" watchObservedRunningTime="2026-09-08 11:55:53.53651864 +0000 UTC m=+1542.395705955" Sep 08 11:55:53.553861 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:53.553839 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk"] Sep 08 11:55:53.556292 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:53.556273 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-489cdd-6796d98d87-wdfsk"] Sep 08 11:55:53.658594 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:53.658559 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a0ea850-5f36-482b-9719-58c6f16a395e" path="/var/lib/kubelet/pods/5a0ea850-5f36-482b-9719-58c6f16a395e/volumes" Sep 08 11:55:59.532543 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:55:59.532511 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:56:05.532526 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:05.532486 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 11:56:05.533924 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:05.532603 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" podUID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 11:56:06.602896 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:06.602865 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" containerID="a27ef6a61230cc204756bd3ec929c9750620b9a2b82670fd1fa59c081c6c93f7" exitCode=0 Sep 08 11:56:06.603294 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:06.602946 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" event={"ID":"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba","Type":"ContainerDied","Data":"a27ef6a61230cc204756bd3ec929c9750620b9a2b82670fd1fa59c081c6c93f7"} Sep 08 11:56:06.603449 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:06.603434 2564 scope.go:117] "RemoveContainer" containerID="a27ef6a61230cc204756bd3ec929c9750620b9a2b82670fd1fa59c081c6c93f7" Sep 08 11:56:06.604551 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:06.604523 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:56:06.604630 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:06.604584 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:56:06.604630 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:06.604626 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 11:56:07.335203 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:07.335168 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:56:07.613752 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:07.613659 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" event={"ID":"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba","Type":"ContainerStarted","Data":"724b46261666ec84f68434005c231c2e261b9242a5ec14507f602922d6913f79"} Sep 08 11:56:08.423846 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:08.423808 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:56:08.424022 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:08.424003 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 08 11:56:08.424075 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:08.424052 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" podUID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 08 11:56:09.424073 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:09.424034 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 08 11:56:09.424496 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:09.424092 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" podUID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 08 11:56:10.424155 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:10.424116 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 08 11:56:10.424570 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:10.424174 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" podUID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 08 11:56:11.424350 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:11.424311 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 08 11:56:11.424757 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:11.424395 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" podUID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 08 11:56:12.424385 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:12.424317 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Sep 08 11:56:12.424773 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:12.424445 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" podUID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Sep 08 11:56:13.427612 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:13.427580 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:56:13.428158 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:13.427858 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:56:13.428611 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:13.428596 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:56:22.197464 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.197428 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp"] Sep 08 11:56:22.197897 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.197636 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" podUID="829bbc4f-fc2d-4db0-9354-ca6acc2c05fe" containerName="maas-api" containerID="cri-o://5b137e4677d6187af5b20c54215a19689b60de8e4f6f7210513b69a2bd3b2feb" gracePeriod=30 Sep 08 11:56:22.451449 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.451390 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:56:22.548571 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.548540 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-metrics-certs\") pod \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " Sep 08 11:56:22.548747 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.548581 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxhgq\" (UniqueName: \"kubernetes.io/projected/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-kube-api-access-mxhgq\") pod \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " Sep 08 11:56:22.548747 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.548640 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-openshift-service-ca\") pod \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " Sep 08 11:56:22.548747 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.548656 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-maas-api-tls\") pod \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\" (UID: \"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe\") " Sep 08 11:56:22.549130 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.549091 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "829bbc4f-fc2d-4db0-9354-ca6acc2c05fe" (UID: "829bbc4f-fc2d-4db0-9354-ca6acc2c05fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:56:22.550927 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.550904 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "829bbc4f-fc2d-4db0-9354-ca6acc2c05fe" (UID: "829bbc4f-fc2d-4db0-9354-ca6acc2c05fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:56:22.551196 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.551181 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "829bbc4f-fc2d-4db0-9354-ca6acc2c05fe" (UID: "829bbc4f-fc2d-4db0-9354-ca6acc2c05fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:56:22.551270 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.551240 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-kube-api-access-mxhgq" (OuterVolumeSpecName: "kube-api-access-mxhgq") pod "829bbc4f-fc2d-4db0-9354-ca6acc2c05fe" (UID: "829bbc4f-fc2d-4db0-9354-ca6acc2c05fe"). InnerVolumeSpecName "kube-api-access-mxhgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:56:22.649781 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.649746 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-openshift-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:22.649781 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.649774 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-maas-api-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:22.649781 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.649784 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-metrics-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:22.650024 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.649792 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxhgq\" (UniqueName: \"kubernetes.io/projected/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe-kube-api-access-mxhgq\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:22.718030 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.717945 2564 generic.go:356] "Generic (PLEG): container finished" podID="829bbc4f-fc2d-4db0-9354-ca6acc2c05fe" containerID="5b137e4677d6187af5b20c54215a19689b60de8e4f6f7210513b69a2bd3b2feb" exitCode=0 Sep 08 11:56:22.718030 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.717987 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" Sep 08 11:56:22.718204 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.718033 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" event={"ID":"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe","Type":"ContainerDied","Data":"5b137e4677d6187af5b20c54215a19689b60de8e4f6f7210513b69a2bd3b2feb"} Sep 08 11:56:22.718204 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.718088 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp" event={"ID":"829bbc4f-fc2d-4db0-9354-ca6acc2c05fe","Type":"ContainerDied","Data":"6c38fcb4c4ce2ab5ba174792b6d0e18fe949d2194501a070623e1ba2122bfe8b"} Sep 08 11:56:22.718204 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.718110 2564 scope.go:117] "RemoveContainer" containerID="5b137e4677d6187af5b20c54215a19689b60de8e4f6f7210513b69a2bd3b2feb" Sep 08 11:56:22.727470 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.727454 2564 scope.go:117] "RemoveContainer" containerID="5b137e4677d6187af5b20c54215a19689b60de8e4f6f7210513b69a2bd3b2feb" Sep 08 11:56:22.727729 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:56:22.727710 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b137e4677d6187af5b20c54215a19689b60de8e4f6f7210513b69a2bd3b2feb\": container with ID starting with 5b137e4677d6187af5b20c54215a19689b60de8e4f6f7210513b69a2bd3b2feb not found: ID does not exist" containerID="5b137e4677d6187af5b20c54215a19689b60de8e4f6f7210513b69a2bd3b2feb" Sep 08 11:56:22.727784 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.727738 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b137e4677d6187af5b20c54215a19689b60de8e4f6f7210513b69a2bd3b2feb"} err="failed to get container status \"5b137e4677d6187af5b20c54215a19689b60de8e4f6f7210513b69a2bd3b2feb\": rpc error: code = NotFound desc = could not find container \"5b137e4677d6187af5b20c54215a19689b60de8e4f6f7210513b69a2bd3b2feb\": container with ID starting with 5b137e4677d6187af5b20c54215a19689b60de8e4f6f7210513b69a2bd3b2feb not found: ID does not exist" Sep 08 11:56:22.740941 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.740910 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp"] Sep 08 11:56:22.745481 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:22.745456 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-64178b-65d9cbf6b9-6qndp"] Sep 08 11:56:23.655084 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:23.655044 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="829bbc4f-fc2d-4db0-9354-ca6acc2c05fe" path="/var/lib/kubelet/pods/829bbc4f-fc2d-4db0-9354-ca6acc2c05fe/volumes" Sep 08 11:56:26.138051 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:26.138019 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:56:29.481158 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:29.481125 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv"] Sep 08 11:56:29.481676 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:29.481335 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" podUID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" containerName="istio-proxy" containerID="cri-o://724b46261666ec84f68434005c231c2e261b9242a5ec14507f602922d6913f79" gracePeriod=30 Sep 08 11:56:34.741207 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.741183 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:56:34.800618 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.800538 2564 generic.go:356] "Generic (PLEG): container finished" podID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" containerID="724b46261666ec84f68434005c231c2e261b9242a5ec14507f602922d6913f79" exitCode=0 Sep 08 11:56:34.800618 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.800606 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" Sep 08 11:56:34.800795 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.800621 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" event={"ID":"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba","Type":"ContainerDied","Data":"724b46261666ec84f68434005c231c2e261b9242a5ec14507f602922d6913f79"} Sep 08 11:56:34.800795 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.800676 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv" event={"ID":"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba","Type":"ContainerDied","Data":"d6d2a97aef8f936baceb810798e56c3e8dd7b5c8ec0953a2ad2a5c2c1fd96a57"} Sep 08 11:56:34.800795 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.800705 2564 scope.go:117] "RemoveContainer" containerID="724b46261666ec84f68434005c231c2e261b9242a5ec14507f602922d6913f79" Sep 08 11:56:34.810132 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.810117 2564 scope.go:117] "RemoveContainer" containerID="a27ef6a61230cc204756bd3ec929c9750620b9a2b82670fd1fa59c081c6c93f7" Sep 08 11:56:34.818903 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.818883 2564 scope.go:117] "RemoveContainer" containerID="724b46261666ec84f68434005c231c2e261b9242a5ec14507f602922d6913f79" Sep 08 11:56:34.819195 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:56:34.819176 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724b46261666ec84f68434005c231c2e261b9242a5ec14507f602922d6913f79\": container with ID starting with 724b46261666ec84f68434005c231c2e261b9242a5ec14507f602922d6913f79 not found: ID does not exist" containerID="724b46261666ec84f68434005c231c2e261b9242a5ec14507f602922d6913f79" Sep 08 11:56:34.819261 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.819203 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724b46261666ec84f68434005c231c2e261b9242a5ec14507f602922d6913f79"} err="failed to get container status \"724b46261666ec84f68434005c231c2e261b9242a5ec14507f602922d6913f79\": rpc error: code = NotFound desc = could not find container \"724b46261666ec84f68434005c231c2e261b9242a5ec14507f602922d6913f79\": container with ID starting with 724b46261666ec84f68434005c231c2e261b9242a5ec14507f602922d6913f79 not found: ID does not exist" Sep 08 11:56:34.819261 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.819220 2564 scope.go:117] "RemoveContainer" containerID="a27ef6a61230cc204756bd3ec929c9750620b9a2b82670fd1fa59c081c6c93f7" Sep 08 11:56:34.819480 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:56:34.819464 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27ef6a61230cc204756bd3ec929c9750620b9a2b82670fd1fa59c081c6c93f7\": container with ID starting with a27ef6a61230cc204756bd3ec929c9750620b9a2b82670fd1fa59c081c6c93f7 not found: ID does not exist" containerID="a27ef6a61230cc204756bd3ec929c9750620b9a2b82670fd1fa59c081c6c93f7" Sep 08 11:56:34.819541 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.819485 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27ef6a61230cc204756bd3ec929c9750620b9a2b82670fd1fa59c081c6c93f7"} err="failed to get container status \"a27ef6a61230cc204756bd3ec929c9750620b9a2b82670fd1fa59c081c6c93f7\": rpc error: code = NotFound desc = could not find container \"a27ef6a61230cc204756bd3ec929c9750620b9a2b82670fd1fa59c081c6c93f7\": container with ID starting with a27ef6a61230cc204756bd3ec929c9750620b9a2b82670fd1fa59c081c6c93f7 not found: ID does not exist" Sep 08 11:56:34.863063 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863040 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-podinfo\") pod \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " Sep 08 11:56:34.863163 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863099 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-credential-socket\") pod \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " Sep 08 11:56:34.863163 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863127 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-token\") pod \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " Sep 08 11:56:34.863258 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863238 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-data\") pod \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " Sep 08 11:56:34.863304 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863284 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6zcn\" (UniqueName: \"kubernetes.io/projected/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-kube-api-access-s6zcn\") pod \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " Sep 08 11:56:34.863354 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863332 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-workload-socket\") pod \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " Sep 08 11:56:34.863429 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863417 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-workload-certs\") pod \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " Sep 08 11:56:34.863488 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863427 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" (UID: "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:56:34.863551 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863472 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istiod-ca-cert\") pod \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " Sep 08 11:56:34.863551 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863536 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-envoy\") pod \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\" (UID: \"7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba\") " Sep 08 11:56:34.863551 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863529 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" (UID: "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:56:34.863697 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863606 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" (UID: "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:56:34.863895 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863871 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" (UID: "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:56:34.863986 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863928 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-credential-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:34.863986 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863948 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-data\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:34.863986 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863962 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-workload-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:34.864143 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.863986 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" (UID: "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:56:34.865252 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.865227 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" (UID: "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 11:56:34.865363 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.865351 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" (UID: "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:56:34.865470 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.865444 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-kube-api-access-s6zcn" (OuterVolumeSpecName: "kube-api-access-s6zcn") pod "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" (UID: "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba"). InnerVolumeSpecName "kube-api-access-s6zcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:56:34.865658 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.865643 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" (UID: "7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:56:34.965187 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.965154 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-workload-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:34.965187 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.965184 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istiod-ca-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:34.965187 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.965196 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-envoy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:34.965454 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.965205 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-podinfo\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:34.965454 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.965213 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:34.965454 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:34.965222 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6zcn\" (UniqueName: \"kubernetes.io/projected/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba-kube-api-access-s6zcn\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:35.125793 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:35.125765 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv"] Sep 08 11:56:35.132617 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:35.132592 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-64178b-openshift-default-6cd7cf5b96-7r4rv"] Sep 08 11:56:35.653864 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:35.653833 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" path="/var/lib/kubelet/pods/7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba/volumes" Sep 08 11:56:48.863399 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:48.863351 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w"] Sep 08 11:56:48.863938 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:48.863602 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" podUID="daad897f-a8af-449f-9176-998ff78f2535" containerName="maas-api" containerID="cri-o://a54b5304db1b8589fbd55bd76fa524cc8cf940fff5f528f7b86f2f62a7df18b3" gracePeriod=30 Sep 08 11:56:49.113323 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.113300 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:56:49.303510 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.303475 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daad897f-a8af-449f-9176-998ff78f2535-openshift-service-ca\") pod \"daad897f-a8af-449f-9176-998ff78f2535\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " Sep 08 11:56:49.303699 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.303535 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad897f-a8af-449f-9176-998ff78f2535-metrics-certs\") pod \"daad897f-a8af-449f-9176-998ff78f2535\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " Sep 08 11:56:49.303699 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.303588 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgmbw\" (UniqueName: \"kubernetes.io/projected/daad897f-a8af-449f-9176-998ff78f2535-kube-api-access-qgmbw\") pod \"daad897f-a8af-449f-9176-998ff78f2535\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " Sep 08 11:56:49.303699 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.303639 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daad897f-a8af-449f-9176-998ff78f2535-maas-api-tls\") pod \"daad897f-a8af-449f-9176-998ff78f2535\" (UID: \"daad897f-a8af-449f-9176-998ff78f2535\") " Sep 08 11:56:49.303875 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.303823 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daad897f-a8af-449f-9176-998ff78f2535-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "daad897f-a8af-449f-9176-998ff78f2535" (UID: "daad897f-a8af-449f-9176-998ff78f2535"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:56:49.305714 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.305680 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daad897f-a8af-449f-9176-998ff78f2535-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "daad897f-a8af-449f-9176-998ff78f2535" (UID: "daad897f-a8af-449f-9176-998ff78f2535"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:56:49.305827 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.305710 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daad897f-a8af-449f-9176-998ff78f2535-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "daad897f-a8af-449f-9176-998ff78f2535" (UID: "daad897f-a8af-449f-9176-998ff78f2535"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:56:49.305827 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.305730 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daad897f-a8af-449f-9176-998ff78f2535-kube-api-access-qgmbw" (OuterVolumeSpecName: "kube-api-access-qgmbw") pod "daad897f-a8af-449f-9176-998ff78f2535" (UID: "daad897f-a8af-449f-9176-998ff78f2535"). InnerVolumeSpecName "kube-api-access-qgmbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:56:49.352709 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.352679 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:56:49.404522 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.404497 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daad897f-a8af-449f-9176-998ff78f2535-openshift-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:49.404522 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.404518 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad897f-a8af-449f-9176-998ff78f2535-metrics-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:49.404522 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.404529 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgmbw\" (UniqueName: \"kubernetes.io/projected/daad897f-a8af-449f-9176-998ff78f2535-kube-api-access-qgmbw\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:49.404712 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.404537 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daad897f-a8af-449f-9176-998ff78f2535-maas-api-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:56:49.894232 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.894198 2564 generic.go:356] "Generic (PLEG): container finished" podID="daad897f-a8af-449f-9176-998ff78f2535" containerID="a54b5304db1b8589fbd55bd76fa524cc8cf940fff5f528f7b86f2f62a7df18b3" exitCode=0 Sep 08 11:56:49.894741 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.894253 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" Sep 08 11:56:49.894741 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.894251 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" event={"ID":"daad897f-a8af-449f-9176-998ff78f2535","Type":"ContainerDied","Data":"a54b5304db1b8589fbd55bd76fa524cc8cf940fff5f528f7b86f2f62a7df18b3"} Sep 08 11:56:49.894741 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.894306 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w" event={"ID":"daad897f-a8af-449f-9176-998ff78f2535","Type":"ContainerDied","Data":"93e5165a0a65cd9fb46d071f5bc5035177c9b34230f39a689df57147f8ca30a9"} Sep 08 11:56:49.894741 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.894327 2564 scope.go:117] "RemoveContainer" containerID="a54b5304db1b8589fbd55bd76fa524cc8cf940fff5f528f7b86f2f62a7df18b3" Sep 08 11:56:49.903199 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.903181 2564 scope.go:117] "RemoveContainer" containerID="a54b5304db1b8589fbd55bd76fa524cc8cf940fff5f528f7b86f2f62a7df18b3" Sep 08 11:56:49.903474 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:56:49.903450 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54b5304db1b8589fbd55bd76fa524cc8cf940fff5f528f7b86f2f62a7df18b3\": container with ID starting with a54b5304db1b8589fbd55bd76fa524cc8cf940fff5f528f7b86f2f62a7df18b3 not found: ID does not exist" containerID="a54b5304db1b8589fbd55bd76fa524cc8cf940fff5f528f7b86f2f62a7df18b3" Sep 08 11:56:49.903596 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.903487 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54b5304db1b8589fbd55bd76fa524cc8cf940fff5f528f7b86f2f62a7df18b3"} err="failed to get container status \"a54b5304db1b8589fbd55bd76fa524cc8cf940fff5f528f7b86f2f62a7df18b3\": rpc error: code = NotFound desc = could not find container \"a54b5304db1b8589fbd55bd76fa524cc8cf940fff5f528f7b86f2f62a7df18b3\": container with ID starting with a54b5304db1b8589fbd55bd76fa524cc8cf940fff5f528f7b86f2f62a7df18b3 not found: ID does not exist" Sep 08 11:56:49.916591 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.916540 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w"] Sep 08 11:56:49.921171 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:49.921151 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3f9288-655dbd67fb-ztz2w"] Sep 08 11:56:51.653031 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:51.652997 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daad897f-a8af-449f-9176-998ff78f2535" path="/var/lib/kubelet/pods/daad897f-a8af-449f-9176-998ff78f2535/volumes" Sep 08 11:56:59.138615 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:56:59.138580 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:57:40.672985 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:40.672952 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25"] Sep 08 11:57:40.673511 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:40.673151 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" podUID="bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3" containerName="maas-api" containerID="cri-o://5b662c52274c675090feaa8490960af7735c6b024d98c331603e0040cf5c6413" gracePeriod=30 Sep 08 11:57:40.929075 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:40.929023 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:57:40.982890 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:40.982861 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-metrics-certs\") pod \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " Sep 08 11:57:40.983062 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:40.982905 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-maas-api-tls\") pod \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " Sep 08 11:57:40.983062 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:40.982924 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcbgc\" (UniqueName: \"kubernetes.io/projected/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-kube-api-access-fcbgc\") pod \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " Sep 08 11:57:40.983175 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:40.983101 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-openshift-service-ca\") pod \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\" (UID: \"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3\") " Sep 08 11:57:40.983590 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:40.983559 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3" (UID: "bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:57:40.985068 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:40.985036 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-kube-api-access-fcbgc" (OuterVolumeSpecName: "kube-api-access-fcbgc") pod "bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3" (UID: "bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3"). InnerVolumeSpecName "kube-api-access-fcbgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:57:40.985177 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:40.985105 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3" (UID: "bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:57:40.985177 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:40.985119 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3" (UID: "bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 11:57:41.084302 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:41.084269 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-maas-api-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:57:41.084302 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:41.084302 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcbgc\" (UniqueName: \"kubernetes.io/projected/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-kube-api-access-fcbgc\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:57:41.084544 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:41.084320 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-openshift-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:57:41.084544 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:41.084333 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3-metrics-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:57:41.196936 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:41.196850 2564 generic.go:356] "Generic (PLEG): container finished" podID="bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3" containerID="5b662c52274c675090feaa8490960af7735c6b024d98c331603e0040cf5c6413" exitCode=0 Sep 08 11:57:41.196936 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:41.196896 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" Sep 08 11:57:41.197129 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:41.196932 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" event={"ID":"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3","Type":"ContainerDied","Data":"5b662c52274c675090feaa8490960af7735c6b024d98c331603e0040cf5c6413"} Sep 08 11:57:41.197129 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:41.196983 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25" event={"ID":"bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3","Type":"ContainerDied","Data":"d427c60993a6f1c6968177264b590ad8993424aa8c209397ac1006b6ad65ea40"} Sep 08 11:57:41.197129 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:41.197006 2564 scope.go:117] "RemoveContainer" containerID="5b662c52274c675090feaa8490960af7735c6b024d98c331603e0040cf5c6413" Sep 08 11:57:41.207058 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:41.207040 2564 scope.go:117] "RemoveContainer" containerID="5b662c52274c675090feaa8490960af7735c6b024d98c331603e0040cf5c6413" Sep 08 11:57:41.207332 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:57:41.207313 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b662c52274c675090feaa8490960af7735c6b024d98c331603e0040cf5c6413\": container with ID starting with 5b662c52274c675090feaa8490960af7735c6b024d98c331603e0040cf5c6413 not found: ID does not exist" containerID="5b662c52274c675090feaa8490960af7735c6b024d98c331603e0040cf5c6413" Sep 08 11:57:41.207434 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:41.207340 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b662c52274c675090feaa8490960af7735c6b024d98c331603e0040cf5c6413"} err="failed to get container status \"5b662c52274c675090feaa8490960af7735c6b024d98c331603e0040cf5c6413\": rpc error: code = NotFound desc = could not find container \"5b662c52274c675090feaa8490960af7735c6b024d98c331603e0040cf5c6413\": container with ID starting with 5b662c52274c675090feaa8490960af7735c6b024d98c331603e0040cf5c6413 not found: ID does not exist" Sep 08 11:57:41.223808 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:41.223784 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25"] Sep 08 11:57:41.228454 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:41.228433 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2ebeba-6dbc64dc68-l4j25"] Sep 08 11:57:41.653586 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:41.653554 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3" path="/var/lib/kubelet/pods/bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3/volumes" Sep 08 11:57:44.262170 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:44.262135 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:57:48.708092 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708057 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4"] Sep 08 11:57:48.708590 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708474 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="829bbc4f-fc2d-4db0-9354-ca6acc2c05fe" containerName="maas-api" Sep 08 11:57:48.708590 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708487 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="829bbc4f-fc2d-4db0-9354-ca6acc2c05fe" containerName="maas-api" Sep 08 11:57:48.708590 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708500 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daad897f-a8af-449f-9176-998ff78f2535" containerName="maas-api" Sep 08 11:57:48.708590 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708507 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="daad897f-a8af-449f-9176-998ff78f2535" containerName="maas-api" Sep 08 11:57:48.708590 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708518 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" containerName="istio-proxy" Sep 08 11:57:48.708590 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708524 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" containerName="istio-proxy" Sep 08 11:57:48.708590 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708536 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" containerName="istio-proxy" Sep 08 11:57:48.708590 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708542 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" containerName="istio-proxy" Sep 08 11:57:48.708590 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708548 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3" containerName="maas-api" Sep 08 11:57:48.708590 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708553 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3" containerName="maas-api" Sep 08 11:57:48.708590 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708564 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0ea850-5f36-482b-9719-58c6f16a395e" containerName="maas-api" Sep 08 11:57:48.708590 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708568 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0ea850-5f36-482b-9719-58c6f16a395e" containerName="maas-api" Sep 08 11:57:48.708972 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708624 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a0ea850-5f36-482b-9719-58c6f16a395e" containerName="maas-api" Sep 08 11:57:48.708972 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708630 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" containerName="istio-proxy" Sep 08 11:57:48.708972 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708638 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c8fb99c-cdee-4ed4-b371-d7ebd1de73ba" containerName="istio-proxy" Sep 08 11:57:48.708972 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708643 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="829bbc4f-fc2d-4db0-9354-ca6acc2c05fe" containerName="maas-api" Sep 08 11:57:48.708972 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708649 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="daad897f-a8af-449f-9176-998ff78f2535" containerName="maas-api" Sep 08 11:57:48.708972 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.708656 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe0571a-9e6f-4d0a-9f5f-1e274f7ad4f3" containerName="maas-api" Sep 08 11:57:48.711804 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.711784 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4" Sep 08 11:57:48.715121 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.715095 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 11:57:48.715842 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.715826 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\"" Sep 08 11:57:48.722981 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.722958 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4"] Sep 08 11:57:48.756985 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.756955 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd43117-96ea-47bf-ada8-711afa6a2e0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4\" (UID: \"4bd43117-96ea-47bf-ada8-711afa6a2e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4" Sep 08 11:57:48.858245 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.858214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd43117-96ea-47bf-ada8-711afa6a2e0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4\" (UID: \"4bd43117-96ea-47bf-ada8-711afa6a2e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4" Sep 08 11:57:48.858831 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:48.858812 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd43117-96ea-47bf-ada8-711afa6a2e0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4\" (UID: \"4bd43117-96ea-47bf-ada8-711afa6a2e0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4" Sep 08 11:57:49.023033 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:49.022946 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4" Sep 08 11:57:49.297936 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:49.297913 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4"] Sep 08 11:57:49.299437 ip-10-0-133-187 kubenswrapper[2564]: W0908 11:57:49.299410 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd43117_96ea_47bf_ada8_711afa6a2e0d.slice/crio-2d8b1f197740c46127062ce1efa09e4b5a913f7b49ab4dc7d5946de2d5623b12 WatchSource:0}: Error finding container 2d8b1f197740c46127062ce1efa09e4b5a913f7b49ab4dc7d5946de2d5623b12: Status 404 returned error can't find the container with id 2d8b1f197740c46127062ce1efa09e4b5a913f7b49ab4dc7d5946de2d5623b12 Sep 08 11:57:50.252691 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:50.252654 2564 generic.go:356] "Generic (PLEG): container finished" podID="4bd43117-96ea-47bf-ada8-711afa6a2e0d" containerID="0a3a432ed5ce9452cecc83d150ee3605a83c1d7e1875c538bcba7b4cdabb9e82" exitCode=0 Sep 08 11:57:50.253105 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:50.252739 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4" event={"ID":"4bd43117-96ea-47bf-ada8-711afa6a2e0d","Type":"ContainerDied","Data":"0a3a432ed5ce9452cecc83d150ee3605a83c1d7e1875c538bcba7b4cdabb9e82"} Sep 08 11:57:50.253105 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:50.252806 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4" event={"ID":"4bd43117-96ea-47bf-ada8-711afa6a2e0d","Type":"ContainerStarted","Data":"2d8b1f197740c46127062ce1efa09e4b5a913f7b49ab4dc7d5946de2d5623b12"} Sep 08 11:57:51.388537 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:51.388511 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4" Sep 08 11:57:51.482995 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:51.482959 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd43117-96ea-47bf-ada8-711afa6a2e0d-maas-api-service-ca\") pod \"4bd43117-96ea-47bf-ada8-711afa6a2e0d\" (UID: \"4bd43117-96ea-47bf-ada8-711afa6a2e0d\") " Sep 08 11:57:51.483313 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:51.483292 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd43117-96ea-47bf-ada8-711afa6a2e0d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4bd43117-96ea-47bf-ada8-711afa6a2e0d" (UID: "4bd43117-96ea-47bf-ada8-711afa6a2e0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:57:51.584499 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:51.584414 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bd43117-96ea-47bf-ada8-711afa6a2e0d-maas-api-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:57:52.264649 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:52.264621 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4" Sep 08 11:57:52.264802 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:52.264619 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4" event={"ID":"4bd43117-96ea-47bf-ada8-711afa6a2e0d","Type":"ContainerDied","Data":"2d8b1f197740c46127062ce1efa09e4b5a913f7b49ab4dc7d5946de2d5623b12"} Sep 08 11:57:52.264802 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:52.264745 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d8b1f197740c46127062ce1efa09e4b5a913f7b49ab4dc7d5946de2d5623b12" Sep 08 11:57:59.546318 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:59.546284 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4"] Sep 08 11:57:59.548673 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:59.548658 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-838a68-hz5b4"] Sep 08 11:57:59.653594 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:57:59.653560 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bd43117-96ea-47bf-ada8-711afa6a2e0d" path="/var/lib/kubelet/pods/4bd43117-96ea-47bf-ada8-711afa6a2e0d/volumes" Sep 08 11:58:02.239152 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:02.239115 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:58:07.391510 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:07.391477 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc"] Sep 08 11:58:07.391966 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:07.391758 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" podUID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" containerName="istio-proxy" containerID="cri-o://d3ca3672b02afe91f8e6b2227ce61737916ec0fe4bbd8fdc7bfcd6af618cbc39" gracePeriod=30 Sep 08 11:58:12.642859 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.642835 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:58:12.787088 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.786998 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjpqz\" (UniqueName: \"kubernetes.io/projected/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-kube-api-access-xjpqz\") pod \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " Sep 08 11:58:12.787088 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787050 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-podinfo\") pod \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " Sep 08 11:58:12.787088 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787076 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-workload-certs\") pod \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " Sep 08 11:58:12.787393 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787115 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-credential-socket\") pod \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " Sep 08 11:58:12.787393 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787143 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istiod-ca-cert\") pod \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " Sep 08 11:58:12.787393 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787180 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-token\") pod \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " Sep 08 11:58:12.787393 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787215 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-data\") pod \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " Sep 08 11:58:12.787393 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787242 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-workload-socket\") pod \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " Sep 08 11:58:12.787393 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787321 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-envoy\") pod \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\" (UID: \"0c99d63f-464e-48f0-b9c6-3893bb4c96ee\") " Sep 08 11:58:12.787740 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787353 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0c99d63f-464e-48f0-b9c6-3893bb4c96ee" (UID: "0c99d63f-464e-48f0-b9c6-3893bb4c96ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:58:12.787740 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787461 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0c99d63f-464e-48f0-b9c6-3893bb4c96ee" (UID: "0c99d63f-464e-48f0-b9c6-3893bb4c96ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:58:12.787740 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787533 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-data" (OuterVolumeSpecName: "istio-data") pod "0c99d63f-464e-48f0-b9c6-3893bb4c96ee" (UID: "0c99d63f-464e-48f0-b9c6-3893bb4c96ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:58:12.787740 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787600 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0c99d63f-464e-48f0-b9c6-3893bb4c96ee" (UID: "0c99d63f-464e-48f0-b9c6-3893bb4c96ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:58:12.787740 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787703 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-workload-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:12.787740 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787723 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-credential-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:12.787740 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787738 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-data\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:12.788109 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787751 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-workload-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:12.788109 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.787890 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0c99d63f-464e-48f0-b9c6-3893bb4c96ee" (UID: "0c99d63f-464e-48f0-b9c6-3893bb4c96ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:58:12.789420 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.789391 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0c99d63f-464e-48f0-b9c6-3893bb4c96ee" (UID: "0c99d63f-464e-48f0-b9c6-3893bb4c96ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:58:12.789566 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.789541 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c99d63f-464e-48f0-b9c6-3893bb4c96ee" (UID: "0c99d63f-464e-48f0-b9c6-3893bb4c96ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:58:12.789653 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.789633 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0c99d63f-464e-48f0-b9c6-3893bb4c96ee" (UID: "0c99d63f-464e-48f0-b9c6-3893bb4c96ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 11:58:12.789750 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.789736 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-kube-api-access-xjpqz" (OuterVolumeSpecName: "kube-api-access-xjpqz") pod "0c99d63f-464e-48f0-b9c6-3893bb4c96ee" (UID: "0c99d63f-464e-48f0-b9c6-3893bb4c96ee"). InnerVolumeSpecName "kube-api-access-xjpqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:58:12.888245 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.888212 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istiod-ca-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:12.888245 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.888236 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:12.888245 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.888248 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-envoy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:12.888490 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.888261 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjpqz\" (UniqueName: \"kubernetes.io/projected/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-kube-api-access-xjpqz\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:12.888490 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:12.888272 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c99d63f-464e-48f0-b9c6-3893bb4c96ee-istio-podinfo\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:13.386689 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:13.386657 2564 generic.go:356] "Generic (PLEG): container finished" podID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" containerID="d3ca3672b02afe91f8e6b2227ce61737916ec0fe4bbd8fdc7bfcd6af618cbc39" exitCode=0 Sep 08 11:58:13.386911 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:13.386706 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" Sep 08 11:58:13.386911 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:13.386747 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" event={"ID":"0c99d63f-464e-48f0-b9c6-3893bb4c96ee","Type":"ContainerDied","Data":"d3ca3672b02afe91f8e6b2227ce61737916ec0fe4bbd8fdc7bfcd6af618cbc39"} Sep 08 11:58:13.386911 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:13.386810 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc" event={"ID":"0c99d63f-464e-48f0-b9c6-3893bb4c96ee","Type":"ContainerDied","Data":"833a8688d3455ce58348584d99a03f79b5d406fdaea344881211c53e2a85d6af"} Sep 08 11:58:13.386911 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:13.386843 2564 scope.go:117] "RemoveContainer" containerID="d3ca3672b02afe91f8e6b2227ce61737916ec0fe4bbd8fdc7bfcd6af618cbc39" Sep 08 11:58:13.397080 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:13.397061 2564 scope.go:117] "RemoveContainer" containerID="dbb4ad2f6c4a44e7b343fa9925e57f7faef22bd9991060d3d307b2abbccfa9e9" Sep 08 11:58:13.405336 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:13.405318 2564 scope.go:117] "RemoveContainer" containerID="d3ca3672b02afe91f8e6b2227ce61737916ec0fe4bbd8fdc7bfcd6af618cbc39" Sep 08 11:58:13.405632 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:58:13.405611 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3ca3672b02afe91f8e6b2227ce61737916ec0fe4bbd8fdc7bfcd6af618cbc39\": container with ID starting with d3ca3672b02afe91f8e6b2227ce61737916ec0fe4bbd8fdc7bfcd6af618cbc39 not found: ID does not exist" containerID="d3ca3672b02afe91f8e6b2227ce61737916ec0fe4bbd8fdc7bfcd6af618cbc39" Sep 08 11:58:13.405703 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:13.405641 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3ca3672b02afe91f8e6b2227ce61737916ec0fe4bbd8fdc7bfcd6af618cbc39"} err="failed to get container status \"d3ca3672b02afe91f8e6b2227ce61737916ec0fe4bbd8fdc7bfcd6af618cbc39\": rpc error: code = NotFound desc = could not find container \"d3ca3672b02afe91f8e6b2227ce61737916ec0fe4bbd8fdc7bfcd6af618cbc39\": container with ID starting with d3ca3672b02afe91f8e6b2227ce61737916ec0fe4bbd8fdc7bfcd6af618cbc39 not found: ID does not exist" Sep 08 11:58:13.405703 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:13.405659 2564 scope.go:117] "RemoveContainer" containerID="dbb4ad2f6c4a44e7b343fa9925e57f7faef22bd9991060d3d307b2abbccfa9e9" Sep 08 11:58:13.405905 ip-10-0-133-187 kubenswrapper[2564]: E0908 11:58:13.405886 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb4ad2f6c4a44e7b343fa9925e57f7faef22bd9991060d3d307b2abbccfa9e9\": container with ID starting with dbb4ad2f6c4a44e7b343fa9925e57f7faef22bd9991060d3d307b2abbccfa9e9 not found: ID does not exist" containerID="dbb4ad2f6c4a44e7b343fa9925e57f7faef22bd9991060d3d307b2abbccfa9e9" Sep 08 11:58:13.405964 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:13.405911 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb4ad2f6c4a44e7b343fa9925e57f7faef22bd9991060d3d307b2abbccfa9e9"} err="failed to get container status \"dbb4ad2f6c4a44e7b343fa9925e57f7faef22bd9991060d3d307b2abbccfa9e9\": rpc error: code = NotFound desc = could not find container \"dbb4ad2f6c4a44e7b343fa9925e57f7faef22bd9991060d3d307b2abbccfa9e9\": container with ID starting with dbb4ad2f6c4a44e7b343fa9925e57f7faef22bd9991060d3d307b2abbccfa9e9 not found: ID does not exist" Sep 08 11:58:13.414065 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:13.414044 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc"] Sep 08 11:58:13.417899 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:13.417877 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-838a68-openshift-default-d7777b99b-glbxc"] Sep 08 11:58:13.653778 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:13.653702 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" path="/var/lib/kubelet/pods/0c99d63f-464e-48f0-b9c6-3893bb4c96ee/volumes" Sep 08 11:58:20.239551 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:20.239513 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:58:26.356110 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:26.356074 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv"] Sep 08 11:58:26.356523 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:26.356326 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerName="istio-proxy" containerID="cri-o://6b4f2f4c0acb85ffe53f63b66152119750473df2ae932457abe00e2a1d787b00" gracePeriod=30 Sep 08 11:58:30.833934 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:30.833891 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 11:58:30.834441 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:30.833963 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 11:58:31.490262 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.490223 2564 generic.go:356] "Generic (PLEG): container finished" podID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerID="6b4f2f4c0acb85ffe53f63b66152119750473df2ae932457abe00e2a1d787b00" exitCode=0 Sep 08 11:58:31.490432 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.490292 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" event={"ID":"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff","Type":"ContainerDied","Data":"6b4f2f4c0acb85ffe53f63b66152119750473df2ae932457abe00e2a1d787b00"} Sep 08 11:58:31.490432 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.490350 2564 scope.go:117] "RemoveContainer" containerID="8e0d6905b20f1f7decc23309293ac2b4a027b8eca0dda96653129498384de662" Sep 08 11:58:31.597919 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.597895 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:58:31.649197 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.649168 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-envoy\") pod \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " Sep 08 11:58:31.649388 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.649250 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-data\") pod \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " Sep 08 11:58:31.649388 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.649293 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istiod-ca-cert\") pod \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " Sep 08 11:58:31.649388 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.649324 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-token\") pod \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " Sep 08 11:58:31.649388 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.649357 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-workload-certs\") pod \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " Sep 08 11:58:31.649626 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.649439 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-credential-socket\") pod \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " Sep 08 11:58:31.649626 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.649477 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-workload-socket\") pod \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " Sep 08 11:58:31.649626 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.649501 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grl2m\" (UniqueName: \"kubernetes.io/projected/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-kube-api-access-grl2m\") pod \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " Sep 08 11:58:31.649626 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.649549 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-podinfo\") pod \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\" (UID: \"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff\") " Sep 08 11:58:31.649626 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.649576 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-data" (OuterVolumeSpecName: "istio-data") pod "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" (UID: "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:58:31.649893 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.649808 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" (UID: "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:58:31.649893 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.649831 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" (UID: "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 11:58:31.649893 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.649890 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-data\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:31.650059 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.649908 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-credential-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:31.650059 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.650000 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" (UID: "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:58:31.650059 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.650017 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" (UID: "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:58:31.651968 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.651933 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-kube-api-access-grl2m" (OuterVolumeSpecName: "kube-api-access-grl2m") pod "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" (UID: "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff"). InnerVolumeSpecName "kube-api-access-grl2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:58:31.652094 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.652000 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" (UID: "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 11:58:31.652306 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.652280 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" (UID: "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 11:58:31.652540 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.652520 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" (UID: "8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 11:58:31.751358 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.751329 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istiod-ca-cert\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:31.751358 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.751353 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-token\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:31.751358 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.751362 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-workload-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:31.751655 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.751396 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-workload-socket\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:31.751655 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.751408 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grl2m\" (UniqueName: \"kubernetes.io/projected/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-kube-api-access-grl2m\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:31.751655 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.751422 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-podinfo\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:31.751655 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.751430 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff-istio-envoy\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 11:58:31.931909 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:31.931834 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 11:58:32.497753 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:32.497726 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" Sep 08 11:58:32.497919 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:32.497730 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv" event={"ID":"8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff","Type":"ContainerDied","Data":"1e1fe6815df79348f2d9379ddef5eb878ed48da0ac5da907ab40e8405bf1a15e"} Sep 08 11:58:32.497919 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:32.497865 2564 scope.go:117] "RemoveContainer" containerID="6b4f2f4c0acb85ffe53f63b66152119750473df2ae932457abe00e2a1d787b00" Sep 08 11:58:32.525199 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:32.522614 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv"] Sep 08 11:58:32.531683 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:32.531659 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bb9b64-openshift-default-7dbbcd994d-bk9sv"] Sep 08 11:58:33.653757 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:58:33.653725 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" path="/var/lib/kubelet/pods/8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff/volumes" Sep 08 11:59:11.904530 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:59:11.904496 2564 scope.go:117] "RemoveContainer" containerID="196648f3d545599d456e5a420447ed53f9b37468c725a3bcf6b3cf7a036ae1ed" Sep 08 11:59:51.541810 ip-10-0-133-187 kubenswrapper[2564]: I0908 11:59:51.541772 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:00:00.140237 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.140193 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814480-kg7qv"] Sep 08 12:00:00.140817 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.140750 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerName="istio-proxy" Sep 08 12:00:00.140817 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.140773 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerName="istio-proxy" Sep 08 12:00:00.140817 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.140784 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bd43117-96ea-47bf-ada8-711afa6a2e0d" containerName="revoke-keys" Sep 08 12:00:00.140817 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.140793 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bd43117-96ea-47bf-ada8-711afa6a2e0d" containerName="revoke-keys" Sep 08 12:00:00.140817 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.140812 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" containerName="istio-proxy" Sep 08 12:00:00.140817 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.140820 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" containerName="istio-proxy" Sep 08 12:00:00.141197 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.140845 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerName="istio-proxy" Sep 08 12:00:00.141197 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.140853 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerName="istio-proxy" Sep 08 12:00:00.141197 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.140941 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" containerName="istio-proxy" Sep 08 12:00:00.141197 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.140955 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerName="istio-proxy" Sep 08 12:00:00.141197 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.140963 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bd43117-96ea-47bf-ada8-711afa6a2e0d" containerName="revoke-keys" Sep 08 12:00:00.141197 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.140973 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" containerName="istio-proxy" Sep 08 12:00:00.141197 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.140982 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3ef434-9302-44ca-86aa-eeb8c8a9c3ff" containerName="istio-proxy" Sep 08 12:00:00.144322 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.144301 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814480-kg7qv" Sep 08 12:00:00.148464 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.148446 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tvt78\"" Sep 08 12:00:00.153831 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.153814 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814480-kg7qv" Sep 08 12:00:00.160529 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.160508 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814480-kg7qv"] Sep 08 12:00:00.433706 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.433680 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814480-kg7qv"] Sep 08 12:00:00.435774 ip-10-0-133-187 kubenswrapper[2564]: W0908 12:00:00.435739 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6102a3a8_509f_46fa_a0b2_52e02fd9b6bb.slice/crio-8649340b9a50b06de788e307f00da7f0b5189f03e7c90afbc82a0d60ef463d07 WatchSource:0}: Error finding container 8649340b9a50b06de788e307f00da7f0b5189f03e7c90afbc82a0d60ef463d07: Status 404 returned error can't find the container with id 8649340b9a50b06de788e307f00da7f0b5189f03e7c90afbc82a0d60ef463d07 Sep 08 12:00:00.995872 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.995781 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814480-kg7qv" event={"ID":"6102a3a8-509f-46fa-a0b2-52e02fd9b6bb","Type":"ContainerStarted","Data":"c4512eac3b3112af1d4b89016c203fc22aa6629738cdac57cb86308a4830305a"} Sep 08 12:00:00.995872 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:00.995838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814480-kg7qv" event={"ID":"6102a3a8-509f-46fa-a0b2-52e02fd9b6bb","Type":"ContainerStarted","Data":"8649340b9a50b06de788e307f00da7f0b5189f03e7c90afbc82a0d60ef463d07"} Sep 08 12:00:01.015185 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:01.015131 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814480-kg7qv" podStartSLOduration=1.015104851 podStartE2EDuration="1.015104851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 12:00:01.013717891 +0000 UTC m=+1789.872905236" watchObservedRunningTime="2026-09-08 12:00:01.015104851 +0000 UTC m=+1789.874292169" Sep 08 12:00:02.003415 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:02.003357 2564 generic.go:356] "Generic (PLEG): container finished" podID="6102a3a8-509f-46fa-a0b2-52e02fd9b6bb" containerID="c4512eac3b3112af1d4b89016c203fc22aa6629738cdac57cb86308a4830305a" exitCode=0 Sep 08 12:00:02.003415 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:02.003400 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814480-kg7qv" event={"ID":"6102a3a8-509f-46fa-a0b2-52e02fd9b6bb","Type":"ContainerDied","Data":"c4512eac3b3112af1d4b89016c203fc22aa6629738cdac57cb86308a4830305a"} Sep 08 12:00:03.141247 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:03.141222 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814480-kg7qv" Sep 08 12:00:04.016054 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:04.015966 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814480-kg7qv" event={"ID":"6102a3a8-509f-46fa-a0b2-52e02fd9b6bb","Type":"ContainerDied","Data":"8649340b9a50b06de788e307f00da7f0b5189f03e7c90afbc82a0d60ef463d07"} Sep 08 12:00:04.016054 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:04.015987 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814480-kg7qv" Sep 08 12:00:04.016054 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:04.016018 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8649340b9a50b06de788e307f00da7f0b5189f03e7c90afbc82a0d60ef463d07" Sep 08 12:00:11.754689 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:11.754655 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 12:00:11.757304 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:11.757282 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 12:00:15.256530 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:15.256485 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:00:23.539657 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:23.539623 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:00:33.547843 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:33.547805 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:00:41.748589 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:41.748551 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:00:56.545600 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:00:56.545563 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:01:04.535975 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:04.535939 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:01:05.620608 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:05.620490 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c"] Sep 08 12:01:05.625708 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:05.620727 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" podUID="178c18c2-ad8f-433a-bf00-281d77b10d43" containerName="manager" containerID="cri-o://0f8d56fafa64bd211c41296bf067217c36e1beeeb4a48e3b5a24278f34c999b5" gracePeriod=10 Sep 08 12:01:06.874194 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:06.874166 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" Sep 08 12:01:06.911956 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:06.911928 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmhdp\" (UniqueName: \"kubernetes.io/projected/178c18c2-ad8f-433a-bf00-281d77b10d43-kube-api-access-cmhdp\") pod \"178c18c2-ad8f-433a-bf00-281d77b10d43\" (UID: \"178c18c2-ad8f-433a-bf00-281d77b10d43\") " Sep 08 12:01:06.912109 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:06.911971 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/178c18c2-ad8f-433a-bf00-281d77b10d43-extensions-socket-volume\") pod \"178c18c2-ad8f-433a-bf00-281d77b10d43\" (UID: \"178c18c2-ad8f-433a-bf00-281d77b10d43\") " Sep 08 12:01:06.912388 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:06.912350 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178c18c2-ad8f-433a-bf00-281d77b10d43-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "178c18c2-ad8f-433a-bf00-281d77b10d43" (UID: "178c18c2-ad8f-433a-bf00-281d77b10d43"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 12:01:06.914231 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:06.914205 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178c18c2-ad8f-433a-bf00-281d77b10d43-kube-api-access-cmhdp" (OuterVolumeSpecName: "kube-api-access-cmhdp") pod "178c18c2-ad8f-433a-bf00-281d77b10d43" (UID: "178c18c2-ad8f-433a-bf00-281d77b10d43"). InnerVolumeSpecName "kube-api-access-cmhdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 12:01:07.012795 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:07.012717 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmhdp\" (UniqueName: \"kubernetes.io/projected/178c18c2-ad8f-433a-bf00-281d77b10d43-kube-api-access-cmhdp\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 12:01:07.012795 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:07.012742 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/178c18c2-ad8f-433a-bf00-281d77b10d43-extensions-socket-volume\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 12:01:07.376899 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:07.376815 2564 generic.go:356] "Generic (PLEG): container finished" podID="178c18c2-ad8f-433a-bf00-281d77b10d43" containerID="0f8d56fafa64bd211c41296bf067217c36e1beeeb4a48e3b5a24278f34c999b5" exitCode=0 Sep 08 12:01:07.376899 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:07.376856 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" Sep 08 12:01:07.377080 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:07.376901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" event={"ID":"178c18c2-ad8f-433a-bf00-281d77b10d43","Type":"ContainerDied","Data":"0f8d56fafa64bd211c41296bf067217c36e1beeeb4a48e3b5a24278f34c999b5"} Sep 08 12:01:07.377080 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:07.376951 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c" event={"ID":"178c18c2-ad8f-433a-bf00-281d77b10d43","Type":"ContainerDied","Data":"f4384d26e4c88f744f0e8b2b1f423a6bb8590709df80544bc9759d37a5ed35d7"} Sep 08 12:01:07.377080 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:07.376974 2564 scope.go:117] "RemoveContainer" containerID="0f8d56fafa64bd211c41296bf067217c36e1beeeb4a48e3b5a24278f34c999b5" Sep 08 12:01:07.394140 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:07.394122 2564 scope.go:117] "RemoveContainer" containerID="0f8d56fafa64bd211c41296bf067217c36e1beeeb4a48e3b5a24278f34c999b5" Sep 08 12:01:07.394467 ip-10-0-133-187 kubenswrapper[2564]: E0908 12:01:07.394444 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8d56fafa64bd211c41296bf067217c36e1beeeb4a48e3b5a24278f34c999b5\": container with ID starting with 0f8d56fafa64bd211c41296bf067217c36e1beeeb4a48e3b5a24278f34c999b5 not found: ID does not exist" containerID="0f8d56fafa64bd211c41296bf067217c36e1beeeb4a48e3b5a24278f34c999b5" Sep 08 12:01:07.394547 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:07.394478 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8d56fafa64bd211c41296bf067217c36e1beeeb4a48e3b5a24278f34c999b5"} err="failed to get container status \"0f8d56fafa64bd211c41296bf067217c36e1beeeb4a48e3b5a24278f34c999b5\": rpc error: code = NotFound desc = could not find container \"0f8d56fafa64bd211c41296bf067217c36e1beeeb4a48e3b5a24278f34c999b5\": container with ID starting with 0f8d56fafa64bd211c41296bf067217c36e1beeeb4a48e3b5a24278f34c999b5 not found: ID does not exist" Sep 08 12:01:07.405831 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:07.405809 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c"] Sep 08 12:01:07.410299 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:07.410278 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-knz2c"] Sep 08 12:01:07.652705 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:01:07.652672 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="178c18c2-ad8f-433a-bf00-281d77b10d43" path="/var/lib/kubelet/pods/178c18c2-ad8f-433a-bf00-281d77b10d43/volumes" Sep 08 12:02:06.362934 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.362903 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc"] Sep 08 12:02:06.363441 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.363258 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" containerName="istio-proxy" Sep 08 12:02:06.363441 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.363269 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c99d63f-464e-48f0-b9c6-3893bb4c96ee" containerName="istio-proxy" Sep 08 12:02:06.363441 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.363288 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="178c18c2-ad8f-433a-bf00-281d77b10d43" containerName="manager" Sep 08 12:02:06.363441 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.363294 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="178c18c2-ad8f-433a-bf00-281d77b10d43" containerName="manager" Sep 08 12:02:06.363441 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.363304 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6102a3a8-509f-46fa-a0b2-52e02fd9b6bb" containerName="cleanup" Sep 08 12:02:06.363441 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.363310 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6102a3a8-509f-46fa-a0b2-52e02fd9b6bb" containerName="cleanup" Sep 08 12:02:06.363441 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.363379 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="178c18c2-ad8f-433a-bf00-281d77b10d43" containerName="manager" Sep 08 12:02:06.363441 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.363388 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6102a3a8-509f-46fa-a0b2-52e02fd9b6bb" containerName="cleanup" Sep 08 12:02:06.366546 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.366530 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc" Sep 08 12:02:06.370031 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.370010 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-d9c72\"" Sep 08 12:02:06.381355 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.381319 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc"] Sep 08 12:02:06.426766 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.426736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcp7g\" (UniqueName: \"kubernetes.io/projected/ad9fc393-18d5-4ac7-b185-64bae574185c-kube-api-access-fcp7g\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-glxzc\" (UID: \"ad9fc393-18d5-4ac7-b185-64bae574185c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc" Sep 08 12:02:06.426913 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.426780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad9fc393-18d5-4ac7-b185-64bae574185c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-glxzc\" (UID: \"ad9fc393-18d5-4ac7-b185-64bae574185c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc" Sep 08 12:02:06.528134 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.528101 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad9fc393-18d5-4ac7-b185-64bae574185c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-glxzc\" (UID: \"ad9fc393-18d5-4ac7-b185-64bae574185c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc" Sep 08 12:02:06.528289 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.528234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcp7g\" (UniqueName: \"kubernetes.io/projected/ad9fc393-18d5-4ac7-b185-64bae574185c-kube-api-access-fcp7g\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-glxzc\" (UID: \"ad9fc393-18d5-4ac7-b185-64bae574185c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc" Sep 08 12:02:06.528530 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.528504 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad9fc393-18d5-4ac7-b185-64bae574185c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-glxzc\" (UID: \"ad9fc393-18d5-4ac7-b185-64bae574185c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc" Sep 08 12:02:06.538896 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.538877 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcp7g\" (UniqueName: \"kubernetes.io/projected/ad9fc393-18d5-4ac7-b185-64bae574185c-kube-api-access-fcp7g\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-glxzc\" (UID: \"ad9fc393-18d5-4ac7-b185-64bae574185c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc" Sep 08 12:02:06.677036 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.677012 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc" Sep 08 12:02:06.960073 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:06.960046 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc"] Sep 08 12:02:06.960617 ip-10-0-133-187 kubenswrapper[2564]: W0908 12:02:06.960590 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9fc393_18d5_4ac7_b185_64bae574185c.slice/crio-28f4ffec9718e94cb1207953ffcd9f0adea935eeaf8de950cac6d6b3da70fe3b WatchSource:0}: Error finding container 28f4ffec9718e94cb1207953ffcd9f0adea935eeaf8de950cac6d6b3da70fe3b: Status 404 returned error can't find the container with id 28f4ffec9718e94cb1207953ffcd9f0adea935eeaf8de950cac6d6b3da70fe3b Sep 08 12:02:07.714760 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:07.714719 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc" event={"ID":"ad9fc393-18d5-4ac7-b185-64bae574185c","Type":"ContainerStarted","Data":"664de3488e029ae5e172f64dd0f13cf2d8ae257c91f01c63d234a64fefb5a364"} Sep 08 12:02:07.715205 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:07.714911 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc" Sep 08 12:02:07.715205 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:07.714942 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc" event={"ID":"ad9fc393-18d5-4ac7-b185-64bae574185c","Type":"ContainerStarted","Data":"28f4ffec9718e94cb1207953ffcd9f0adea935eeaf8de950cac6d6b3da70fe3b"} Sep 08 12:02:07.742532 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:07.742476 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc" podStartSLOduration=1.742455431 podStartE2EDuration="1.742455431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 12:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 12:02:07.740767406 +0000 UTC m=+1916.599954743" watchObservedRunningTime="2026-09-08 12:02:07.742455431 +0000 UTC m=+1916.601642749" Sep 08 12:02:18.722936 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:18.722907 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-glxzc" Sep 08 12:02:34.434382 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:34.434328 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:02:40.247201 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:40.247160 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:02:50.558979 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:02:50.558945 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:03:12.343113 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:03:12.343035 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:03:44.743960 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:03:44.743916 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:03:52.335827 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:03:52.335796 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:03:57.743996 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:03:57.743960 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:04:02.751852 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.751817 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:04:02.816097 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.816061 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj"] Sep 08 12:04:02.819534 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.819515 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:02.823462 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.823433 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 12:04:02.823577 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.823508 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 08 12:04:02.823577 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.823526 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b5t6p\"" Sep 08 12:04:02.823577 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.823534 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 08 12:04:02.823983 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.823654 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 08 12:04:02.832500 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.832477 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj"] Sep 08 12:04:02.835453 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.835428 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l"] Sep 08 12:04:02.839264 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.839247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:02.850783 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.850763 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l"] Sep 08 12:04:02.888544 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.888521 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acce666e-d4f6-4152-9753-12b4bc44a28e-maas-api-tls\") pod \"maas-api-cf5b5fd48-hgl6l\" (UID: \"acce666e-d4f6-4152-9753-12b4bc44a28e\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:02.888660 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.888587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-maas-api-tls\") pod \"maas-api-7df9d55888-t2qcj\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:02.888660 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.888622 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acce666e-d4f6-4152-9753-12b4bc44a28e-metrics-certs\") pod \"maas-api-cf5b5fd48-hgl6l\" (UID: \"acce666e-d4f6-4152-9753-12b4bc44a28e\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:02.888660 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.888643 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acce666e-d4f6-4152-9753-12b4bc44a28e-openshift-service-ca\") pod \"maas-api-cf5b5fd48-hgl6l\" (UID: \"acce666e-d4f6-4152-9753-12b4bc44a28e\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:02.888765 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.888663 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-openshift-service-ca\") pod \"maas-api-7df9d55888-t2qcj\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:02.888765 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.888740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7scrm\" (UniqueName: \"kubernetes.io/projected/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-kube-api-access-7scrm\") pod \"maas-api-7df9d55888-t2qcj\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:02.888833 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.888778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h8r5\" (UniqueName: \"kubernetes.io/projected/acce666e-d4f6-4152-9753-12b4bc44a28e-kube-api-access-8h8r5\") pod \"maas-api-cf5b5fd48-hgl6l\" (UID: \"acce666e-d4f6-4152-9753-12b4bc44a28e\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:02.888833 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.888803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-metrics-certs\") pod \"maas-api-7df9d55888-t2qcj\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:02.989789 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.989763 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h8r5\" (UniqueName: \"kubernetes.io/projected/acce666e-d4f6-4152-9753-12b4bc44a28e-kube-api-access-8h8r5\") pod \"maas-api-cf5b5fd48-hgl6l\" (UID: \"acce666e-d4f6-4152-9753-12b4bc44a28e\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:02.989918 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.989793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-metrics-certs\") pod \"maas-api-7df9d55888-t2qcj\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:02.989918 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.989825 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acce666e-d4f6-4152-9753-12b4bc44a28e-maas-api-tls\") pod \"maas-api-cf5b5fd48-hgl6l\" (UID: \"acce666e-d4f6-4152-9753-12b4bc44a28e\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:02.990042 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.989941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-maas-api-tls\") pod \"maas-api-7df9d55888-t2qcj\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:02.990042 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.989987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acce666e-d4f6-4152-9753-12b4bc44a28e-metrics-certs\") pod \"maas-api-cf5b5fd48-hgl6l\" (UID: \"acce666e-d4f6-4152-9753-12b4bc44a28e\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:02.990042 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.990016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acce666e-d4f6-4152-9753-12b4bc44a28e-openshift-service-ca\") pod \"maas-api-cf5b5fd48-hgl6l\" (UID: \"acce666e-d4f6-4152-9753-12b4bc44a28e\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:02.990218 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.990045 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-openshift-service-ca\") pod \"maas-api-7df9d55888-t2qcj\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:02.990218 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.990101 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7scrm\" (UniqueName: \"kubernetes.io/projected/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-kube-api-access-7scrm\") pod \"maas-api-7df9d55888-t2qcj\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:02.990837 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.990798 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acce666e-d4f6-4152-9753-12b4bc44a28e-openshift-service-ca\") pod \"maas-api-cf5b5fd48-hgl6l\" (UID: \"acce666e-d4f6-4152-9753-12b4bc44a28e\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:02.990968 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.990885 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-openshift-service-ca\") pod \"maas-api-7df9d55888-t2qcj\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:02.992505 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.992483 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acce666e-d4f6-4152-9753-12b4bc44a28e-maas-api-tls\") pod \"maas-api-cf5b5fd48-hgl6l\" (UID: \"acce666e-d4f6-4152-9753-12b4bc44a28e\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:02.992626 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.992607 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acce666e-d4f6-4152-9753-12b4bc44a28e-metrics-certs\") pod \"maas-api-cf5b5fd48-hgl6l\" (UID: \"acce666e-d4f6-4152-9753-12b4bc44a28e\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:02.992676 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.992649 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-metrics-certs\") pod \"maas-api-7df9d55888-t2qcj\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:02.992838 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.992816 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-maas-api-tls\") pod \"maas-api-7df9d55888-t2qcj\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:02.999247 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.999225 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7scrm\" (UniqueName: \"kubernetes.io/projected/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-kube-api-access-7scrm\") pod \"maas-api-7df9d55888-t2qcj\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:02.999813 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:02.999793 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h8r5\" (UniqueName: \"kubernetes.io/projected/acce666e-d4f6-4152-9753-12b4bc44a28e-kube-api-access-8h8r5\") pod \"maas-api-cf5b5fd48-hgl6l\" (UID: \"acce666e-d4f6-4152-9753-12b4bc44a28e\") " pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:03.130215 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:03.130149 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:03.149285 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:03.149257 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:03.416644 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:03.416619 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj"] Sep 08 12:04:03.417881 ip-10-0-133-187 kubenswrapper[2564]: W0908 12:04:03.417857 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b17642e_1ba6_4f17_9461_0cb4c78c8fb2.slice/crio-01f7a0f323553b3cb7f531fdabcd6e12ec5630a4203e02a36e7bba9a1ace243c WatchSource:0}: Error finding container 01f7a0f323553b3cb7f531fdabcd6e12ec5630a4203e02a36e7bba9a1ace243c: Status 404 returned error can't find the container with id 01f7a0f323553b3cb7f531fdabcd6e12ec5630a4203e02a36e7bba9a1ace243c Sep 08 12:04:03.419265 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:03.419246 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 12:04:03.432712 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:03.432687 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l"] Sep 08 12:04:03.433148 ip-10-0-133-187 kubenswrapper[2564]: W0908 12:04:03.433126 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacce666e_d4f6_4152_9753_12b4bc44a28e.slice/crio-c2bd21e4196c01726aea254fafc349119c6b97961231c1c089001905729257d3 WatchSource:0}: Error finding container c2bd21e4196c01726aea254fafc349119c6b97961231c1c089001905729257d3: Status 404 returned error can't find the container with id c2bd21e4196c01726aea254fafc349119c6b97961231c1c089001905729257d3 Sep 08 12:04:04.364399 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:04.364289 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" event={"ID":"acce666e-d4f6-4152-9753-12b4bc44a28e","Type":"ContainerStarted","Data":"2113342fefc75632eff1265ae2916b0feed556f33a87b1eae2cc6f4bad9a0b11"} Sep 08 12:04:04.364399 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:04.364352 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" event={"ID":"acce666e-d4f6-4152-9753-12b4bc44a28e","Type":"ContainerStarted","Data":"c2bd21e4196c01726aea254fafc349119c6b97961231c1c089001905729257d3"} Sep 08 12:04:04.364399 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:04.364398 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:04.366114 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:04.366089 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" event={"ID":"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2","Type":"ContainerStarted","Data":"5519df33a979548a6eba3fbb93c2f843ef85808eb9e5693ea2cf5fce809708d9"} Sep 08 12:04:04.366249 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:04.366138 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" event={"ID":"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2","Type":"ContainerStarted","Data":"01f7a0f323553b3cb7f531fdabcd6e12ec5630a4203e02a36e7bba9a1ace243c"} Sep 08 12:04:04.366249 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:04.366174 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:04.385773 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:04.385727 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" podStartSLOduration=1.890429291 podStartE2EDuration="2.385714595s" totalImagesPullingTime="495.285304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 12:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 12:04:04.383430759 +0000 UTC m=+2033.242618098" watchObservedRunningTime="2026-09-08 12:04:04.385714595 +0000 UTC m=+2033.244901913" Sep 08 12:04:04.404831 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:04.404791 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" podStartSLOduration=1.881983414 podStartE2EDuration="2.404776304s" totalImagesPullingTime="522.79289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 12:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 12:04:04.402404392 +0000 UTC m=+2033.261591724" watchObservedRunningTime="2026-09-08 12:04:04.404776304 +0000 UTC m=+2033.263963622" Sep 08 12:04:10.376637 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.376606 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-cf5b5fd48-hgl6l" Sep 08 12:04:10.377057 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.376971 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:10.471413 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.471362 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj"] Sep 08 12:04:10.471636 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.471612 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" podUID="8b17642e-1ba6-4f17-9461-0cb4c78c8fb2" containerName="maas-api" containerID="cri-o://5519df33a979548a6eba3fbb93c2f843ef85808eb9e5693ea2cf5fce809708d9" gracePeriod=30 Sep 08 12:04:10.724801 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.724778 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:10.760252 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.760221 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-openshift-service-ca\") pod \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " Sep 08 12:04:10.760416 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.760260 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-metrics-certs\") pod \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " Sep 08 12:04:10.760416 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.760298 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7scrm\" (UniqueName: \"kubernetes.io/projected/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-kube-api-access-7scrm\") pod \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " Sep 08 12:04:10.760416 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.760344 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-maas-api-tls\") pod \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\" (UID: \"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2\") " Sep 08 12:04:10.760642 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.760613 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b17642e-1ba6-4f17-9461-0cb4c78c8fb2" (UID: "8b17642e-1ba6-4f17-9461-0cb4c78c8fb2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 12:04:10.762431 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.762407 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b17642e-1ba6-4f17-9461-0cb4c78c8fb2" (UID: "8b17642e-1ba6-4f17-9461-0cb4c78c8fb2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 12:04:10.762565 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.762446 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8b17642e-1ba6-4f17-9461-0cb4c78c8fb2" (UID: "8b17642e-1ba6-4f17-9461-0cb4c78c8fb2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 12:04:10.762565 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.762523 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-kube-api-access-7scrm" (OuterVolumeSpecName: "kube-api-access-7scrm") pod "8b17642e-1ba6-4f17-9461-0cb4c78c8fb2" (UID: "8b17642e-1ba6-4f17-9461-0cb4c78c8fb2"). InnerVolumeSpecName "kube-api-access-7scrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 12:04:10.862058 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.862032 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-openshift-service-ca\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 12:04:10.862058 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.862056 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-metrics-certs\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 12:04:10.862226 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.862067 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7scrm\" (UniqueName: \"kubernetes.io/projected/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-kube-api-access-7scrm\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 12:04:10.862226 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:10.862076 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2-maas-api-tls\") on node \"ip-10-0-133-187.ec2.internal\" DevicePath \"\"" Sep 08 12:04:11.407125 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:11.407090 2564 generic.go:356] "Generic (PLEG): container finished" podID="8b17642e-1ba6-4f17-9461-0cb4c78c8fb2" containerID="5519df33a979548a6eba3fbb93c2f843ef85808eb9e5693ea2cf5fce809708d9" exitCode=0 Sep 08 12:04:11.407512 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:11.407139 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" Sep 08 12:04:11.407512 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:11.407180 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" event={"ID":"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2","Type":"ContainerDied","Data":"5519df33a979548a6eba3fbb93c2f843ef85808eb9e5693ea2cf5fce809708d9"} Sep 08 12:04:11.407512 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:11.407241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj" event={"ID":"8b17642e-1ba6-4f17-9461-0cb4c78c8fb2","Type":"ContainerDied","Data":"01f7a0f323553b3cb7f531fdabcd6e12ec5630a4203e02a36e7bba9a1ace243c"} Sep 08 12:04:11.407512 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:11.407270 2564 scope.go:117] "RemoveContainer" containerID="5519df33a979548a6eba3fbb93c2f843ef85808eb9e5693ea2cf5fce809708d9" Sep 08 12:04:11.417174 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:11.417158 2564 scope.go:117] "RemoveContainer" containerID="5519df33a979548a6eba3fbb93c2f843ef85808eb9e5693ea2cf5fce809708d9" Sep 08 12:04:11.417444 ip-10-0-133-187 kubenswrapper[2564]: E0908 12:04:11.417420 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5519df33a979548a6eba3fbb93c2f843ef85808eb9e5693ea2cf5fce809708d9\": container with ID starting with 5519df33a979548a6eba3fbb93c2f843ef85808eb9e5693ea2cf5fce809708d9 not found: ID does not exist" containerID="5519df33a979548a6eba3fbb93c2f843ef85808eb9e5693ea2cf5fce809708d9" Sep 08 12:04:11.417788 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:11.417450 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5519df33a979548a6eba3fbb93c2f843ef85808eb9e5693ea2cf5fce809708d9"} err="failed to get container status \"5519df33a979548a6eba3fbb93c2f843ef85808eb9e5693ea2cf5fce809708d9\": rpc error: code = NotFound desc = could not find container \"5519df33a979548a6eba3fbb93c2f843ef85808eb9e5693ea2cf5fce809708d9\": container with ID starting with 5519df33a979548a6eba3fbb93c2f843ef85808eb9e5693ea2cf5fce809708d9 not found: ID does not exist" Sep 08 12:04:11.433099 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:11.433076 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj"] Sep 08 12:04:11.440279 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:11.440258 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7df9d55888-t2qcj"] Sep 08 12:04:11.653943 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:11.653911 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b17642e-1ba6-4f17-9461-0cb4c78c8fb2" path="/var/lib/kubelet/pods/8b17642e-1ba6-4f17-9461-0cb4c78c8fb2/volumes" Sep 08 12:04:11.850377 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:11.850292 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:04:11.975511 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:11.975479 2564 scope.go:117] "RemoveContainer" containerID="0a3a432ed5ce9452cecc83d150ee3605a83c1d7e1875c538bcba7b4cdabb9e82" Sep 08 12:04:20.971637 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:20.971603 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:04:27.440731 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:27.440694 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:04:34.686391 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:34.686294 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-nn5kg"] Sep 08 12:04:43.780739 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:43.780702 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-r6fln_6e91d826-a41c-4240-ab36-d8f88cf8f746/manager/0.log" Sep 08 12:04:44.509576 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:44.509548 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd45fbd48-q7fmb_c2033805-0e78-4a2e-afe7-d31c4fc4672d/copy-manifests/0.log" Sep 08 12:04:44.529010 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:44.528979 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd45fbd48-q7fmb_c2033805-0e78-4a2e-afe7-d31c4fc4672d/manager/0.log" Sep 08 12:04:45.003250 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:45.003215 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-cf5b5fd48-hgl6l_acce666e-d4f6-4152-9753-12b4bc44a28e/maas-api/0.log" Sep 08 12:04:45.329808 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:45.329727 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814480-kg7qv_6102a3a8-509f-46fa-a0b2-52e02fd9b6bb/cleanup/0.log" Sep 08 12:04:46.335891 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:46.335860 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-ffxcd_70da0468-a13e-4146-a5be-0dd76443cd46/manager/0.log" Sep 08 12:04:46.451765 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:46.451726 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-hrnxj_00cbf4b6-5a38-46f2-9b62-7a5d4088bc9e/manager/0.log" Sep 08 12:04:46.682414 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:46.682380 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-glxzc_ad9fc393-18d5-4ac7-b185-64bae574185c/manager/0.log" Sep 08 12:04:46.794115 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:46.794087 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-nn5kg_b7c28493-f1fd-49bc-a64a-cef03d75acf9/limitador/0.log" Sep 08 12:04:47.393445 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:47.393418 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-frt82_22686641-81b6-402d-b231-2e8662c60034/discovery/0.log" Sep 08 12:04:47.513146 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:47.513116 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8d66d765-ftb49_701c944e-55c1-4c6f-8f9e-164339efbe3d/kube-auth-proxy/0.log" Sep 08 12:04:48.791114 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:48.791085 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-7v499_5752df83-029c-4b2f-bc14-57c2e46646e3/main/0.log" Sep 08 12:04:48.916107 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:48.916077 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6hz7t_73a2e868-7b61-4f6e-81b4-de979eb45479/main/0.log" Sep 08 12:04:57.687246 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:57.687216 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8r82d_74934df5-b10d-497b-9ada-49aae488dd2b/global-pull-secret-syncer/0.log" Sep 08 12:04:57.856441 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:57.856400 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bjthr_445df2c7-2335-4a07-bf1d-ad753e708cc3/konnectivity-agent/0.log" Sep 08 12:04:57.880751 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:04:57.880715 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-187.ec2.internal_18abe9a9a9441b764e0c8f71a4e8bdc5/haproxy/0.log" Sep 08 12:05:02.215797 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:02.215756 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-ffxcd_70da0468-a13e-4146-a5be-0dd76443cd46/manager/0.log" Sep 08 12:05:02.249458 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:02.249430 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-hrnxj_00cbf4b6-5a38-46f2-9b62-7a5d4088bc9e/manager/0.log" Sep 08 12:05:02.332572 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:02.332547 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-glxzc_ad9fc393-18d5-4ac7-b185-64bae574185c/manager/0.log" Sep 08 12:05:02.369132 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:02.369102 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-nn5kg_b7c28493-f1fd-49bc-a64a-cef03d75acf9/limitador/0.log" Sep 08 12:05:04.425447 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:04.425417 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6dd97df8-b75x6_89e99fa4-2f32-4165-883d-228ff3a4e15b/metrics-server/0.log" Sep 08 12:05:04.455660 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:04.455629 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-75bh8_857c1344-c926-43c7-961b-39b71303a7df/monitoring-plugin/0.log" Sep 08 12:05:04.733140 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:04.733056 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x4fkq_4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b/node-exporter/0.log" Sep 08 12:05:04.767228 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:04.767195 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x4fkq_4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b/kube-rbac-proxy/0.log" Sep 08 12:05:04.796420 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:04.796390 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x4fkq_4492f8b8-3d1e-4aaf-9e56-ae877dbffd4b/init-textfile/0.log" Sep 08 12:05:05.247821 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:05.247787 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-r568v_db55a245-eaef-4e08-9bc4-b09356c8db5f/prometheus-operator/0.log" Sep 08 12:05:05.275089 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:05.275063 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-r568v_db55a245-eaef-4e08-9bc4-b09356c8db5f/kube-rbac-proxy/0.log" Sep 08 12:05:05.324846 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:05.324823 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-q6n28_3aca23dd-e56b-4e85-8057-ec729bd673e3/prometheus-operator-admission-webhook/0.log" Sep 08 12:05:05.484847 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:05.484814 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d48c6b94d-s4q8p_63238fe1-9071-4c9d-a037-c25a0bd5560c/thanos-query/0.log" Sep 08 12:05:05.514952 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:05.514871 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d48c6b94d-s4q8p_63238fe1-9071-4c9d-a037-c25a0bd5560c/kube-rbac-proxy-web/0.log" Sep 08 12:05:05.546922 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:05.546893 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d48c6b94d-s4q8p_63238fe1-9071-4c9d-a037-c25a0bd5560c/kube-rbac-proxy/0.log" Sep 08 12:05:05.578947 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:05.578923 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d48c6b94d-s4q8p_63238fe1-9071-4c9d-a037-c25a0bd5560c/prom-label-proxy/0.log" Sep 08 12:05:05.606196 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:05.606166 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d48c6b94d-s4q8p_63238fe1-9071-4c9d-a037-c25a0bd5560c/kube-rbac-proxy-rules/0.log" Sep 08 12:05:05.634481 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:05.634448 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d48c6b94d-s4q8p_63238fe1-9071-4c9d-a037-c25a0bd5560c/kube-rbac-proxy-metrics/0.log" Sep 08 12:05:07.433496 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.433461 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb"] Sep 08 12:05:07.433880 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.433820 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b17642e-1ba6-4f17-9461-0cb4c78c8fb2" containerName="maas-api" Sep 08 12:05:07.433880 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.433831 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b17642e-1ba6-4f17-9461-0cb4c78c8fb2" containerName="maas-api" Sep 08 12:05:07.433953 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.433917 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b17642e-1ba6-4f17-9461-0cb4c78c8fb2" containerName="maas-api" Sep 08 12:05:07.437267 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.437251 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.445007 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.444986 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ghzlp\"/\"default-dockercfg-xv459\"" Sep 08 12:05:07.445125 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.444986 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ghzlp\"/\"openshift-service-ca.crt\"" Sep 08 12:05:07.445125 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.444986 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ghzlp\"/\"kube-root-ca.crt\"" Sep 08 12:05:07.450840 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.450817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e99075cf-1ca8-4781-bea9-594d61be09fc-podres\") pod \"perf-node-gather-daemonset-mrkkb\" (UID: \"e99075cf-1ca8-4781-bea9-594d61be09fc\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.450931 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.450891 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e99075cf-1ca8-4781-bea9-594d61be09fc-proc\") pod \"perf-node-gather-daemonset-mrkkb\" (UID: \"e99075cf-1ca8-4781-bea9-594d61be09fc\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.450931 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.450914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp47g\" (UniqueName: \"kubernetes.io/projected/e99075cf-1ca8-4781-bea9-594d61be09fc-kube-api-access-bp47g\") pod \"perf-node-gather-daemonset-mrkkb\" (UID: \"e99075cf-1ca8-4781-bea9-594d61be09fc\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.451019 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.450932 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e99075cf-1ca8-4781-bea9-594d61be09fc-sys\") pod \"perf-node-gather-daemonset-mrkkb\" (UID: \"e99075cf-1ca8-4781-bea9-594d61be09fc\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.451019 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.450952 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e99075cf-1ca8-4781-bea9-594d61be09fc-lib-modules\") pod \"perf-node-gather-daemonset-mrkkb\" (UID: \"e99075cf-1ca8-4781-bea9-594d61be09fc\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.463097 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.463070 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb"] Sep 08 12:05:07.552019 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.551984 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e99075cf-1ca8-4781-bea9-594d61be09fc-podres\") pod \"perf-node-gather-daemonset-mrkkb\" (UID: \"e99075cf-1ca8-4781-bea9-594d61be09fc\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.552191 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.552063 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e99075cf-1ca8-4781-bea9-594d61be09fc-proc\") pod \"perf-node-gather-daemonset-mrkkb\" (UID: \"e99075cf-1ca8-4781-bea9-594d61be09fc\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.552191 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.552102 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp47g\" (UniqueName: \"kubernetes.io/projected/e99075cf-1ca8-4781-bea9-594d61be09fc-kube-api-access-bp47g\") pod \"perf-node-gather-daemonset-mrkkb\" (UID: \"e99075cf-1ca8-4781-bea9-594d61be09fc\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.552191 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.552121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e99075cf-1ca8-4781-bea9-594d61be09fc-sys\") pod \"perf-node-gather-daemonset-mrkkb\" (UID: \"e99075cf-1ca8-4781-bea9-594d61be09fc\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.552191 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.552137 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e99075cf-1ca8-4781-bea9-594d61be09fc-proc\") pod \"perf-node-gather-daemonset-mrkkb\" (UID: \"e99075cf-1ca8-4781-bea9-594d61be09fc\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.552191 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.552150 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e99075cf-1ca8-4781-bea9-594d61be09fc-podres\") pod \"perf-node-gather-daemonset-mrkkb\" (UID: \"e99075cf-1ca8-4781-bea9-594d61be09fc\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.552191 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.552143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e99075cf-1ca8-4781-bea9-594d61be09fc-lib-modules\") pod \"perf-node-gather-daemonset-mrkkb\" (UID: \"e99075cf-1ca8-4781-bea9-594d61be09fc\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.552430 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.552201 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e99075cf-1ca8-4781-bea9-594d61be09fc-sys\") pod \"perf-node-gather-daemonset-mrkkb\" (UID: \"e99075cf-1ca8-4781-bea9-594d61be09fc\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.552430 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.552236 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e99075cf-1ca8-4781-bea9-594d61be09fc-lib-modules\") pod \"perf-node-gather-daemonset-mrkkb\" (UID: \"e99075cf-1ca8-4781-bea9-594d61be09fc\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.571126 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.571096 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp47g\" (UniqueName: \"kubernetes.io/projected/e99075cf-1ca8-4781-bea9-594d61be09fc-kube-api-access-bp47g\") pod \"perf-node-gather-daemonset-mrkkb\" (UID: \"e99075cf-1ca8-4781-bea9-594d61be09fc\") " pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:07.695386 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.695269 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 12:05:07.703360 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.703339 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/2.log" Sep 08 12:05:07.747601 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:07.747572 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:08.038108 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:08.038083 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb"] Sep 08 12:05:08.039992 ip-10-0-133-187 kubenswrapper[2564]: W0908 12:05:08.039960 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode99075cf_1ca8_4781_bea9_594d61be09fc.slice/crio-c9e029c388315dcc37ec958a5d1e7dd5cb244471d897faae86b30a63f10023ee WatchSource:0}: Error finding container c9e029c388315dcc37ec958a5d1e7dd5cb244471d897faae86b30a63f10023ee: Status 404 returned error can't find the container with id c9e029c388315dcc37ec958a5d1e7dd5cb244471d897faae86b30a63f10023ee Sep 08 12:05:08.391318 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:08.391286 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7f5bc5947f-l2h7n_bba36071-18ff-4c97-8b17-4706495be193/console/0.log" Sep 08 12:05:08.447189 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:08.447163 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-wlp42_8bb30630-55f6-4d76-9406-958da86179d3/download-server/0.log" Sep 08 12:05:08.729991 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:08.729891 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" event={"ID":"e99075cf-1ca8-4781-bea9-594d61be09fc","Type":"ContainerStarted","Data":"3f326897099d55718f51c6fe7c8e81116e4e8800f15075318757483fd1647083"} Sep 08 12:05:08.729991 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:08.729969 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" event={"ID":"e99075cf-1ca8-4781-bea9-594d61be09fc","Type":"ContainerStarted","Data":"c9e029c388315dcc37ec958a5d1e7dd5cb244471d897faae86b30a63f10023ee"} Sep 08 12:05:08.730171 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:08.729995 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:08.754772 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:08.754715 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" podStartSLOduration=1.754700267 podStartE2EDuration="1.754700267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 12:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 12:05:08.754115421 +0000 UTC m=+2097.613302741" watchObservedRunningTime="2026-09-08 12:05:08.754700267 +0000 UTC m=+2097.613887617" Sep 08 12:05:10.397341 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:10.397304 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-66f5s_645f6f42-ec34-4b66-bf8e-183601ddc1af/dns/0.log" Sep 08 12:05:10.425044 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:10.425015 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-66f5s_645f6f42-ec34-4b66-bf8e-183601ddc1af/kube-rbac-proxy/0.log" Sep 08 12:05:10.577864 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:10.577830 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5vslm_15e73439-0f50-4c2e-9155-3e18ec9e5231/dns-node-resolver/0.log" Sep 08 12:05:11.336160 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:11.336131 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sb576_c6a08843-d61b-451e-8e36-9e949a01059d/node-ca/0.log" Sep 08 12:05:11.792949 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:11.792922 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 12:05:11.796362 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:11.796341 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-2mxp7_a3af17fe-667d-4032-9a0e-aaadfcdacb2f/console-operator/1.log" Sep 08 12:05:12.985768 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:12.985737 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-frt82_22686641-81b6-402d-b231-2e8662c60034/discovery/0.log" Sep 08 12:05:13.016568 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:13.016532 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8d66d765-ftb49_701c944e-55c1-4c6f-8f9e-164339efbe3d/kube-auth-proxy/0.log" Sep 08 12:05:14.141663 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:14.141634 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lfp5b_535465a6-a1fc-4d49-917c-6bff81183ae9/serve-healthcheck-canary/0.log" Sep 08 12:05:14.734165 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:14.734129 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-p5jkx_f44bd9c6-f403-43f1-820f-e49f9ded15e6/insights-operator/0.log" Sep 08 12:05:14.734165 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:14.734149 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-p5jkx_f44bd9c6-f403-43f1-820f-e49f9ded15e6/insights-operator/1.log" Sep 08 12:05:14.744299 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:14.744268 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ghzlp/perf-node-gather-daemonset-mrkkb" Sep 08 12:05:14.853505 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:14.853472 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pghh7_3830412b-f8ce-406d-b026-369258aba608/kube-rbac-proxy/0.log" Sep 08 12:05:14.881788 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:14.881767 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pghh7_3830412b-f8ce-406d-b026-369258aba608/exporter/0.log" Sep 08 12:05:14.914155 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:14.914132 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pghh7_3830412b-f8ce-406d-b026-369258aba608/extractor/0.log" Sep 08 12:05:17.441811 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:17.441781 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-r6fln_6e91d826-a41c-4240-ab36-d8f88cf8f746/manager/0.log" Sep 08 12:05:18.106038 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:18.106010 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd45fbd48-q7fmb_c2033805-0e78-4a2e-afe7-d31c4fc4672d/manager/0.log" Sep 08 12:05:18.147057 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:18.147030 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd45fbd48-q7fmb_c2033805-0e78-4a2e-afe7-d31c4fc4672d/copy-manifests/0.log" Sep 08 12:05:18.353777 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:18.353748 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-frt82_22686641-81b6-402d-b231-2e8662c60034/discovery/0.log" Sep 08 12:05:18.372162 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:18.372133 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8d66d765-ftb49_701c944e-55c1-4c6f-8f9e-164339efbe3d/kube-auth-proxy/0.log" Sep 08 12:05:20.610394 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:20.610346 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-frt82_22686641-81b6-402d-b231-2e8662c60034/discovery/0.log" Sep 08 12:05:20.644261 ip-10-0-133-187 kubenswrapper[2564]: I0908 12:05:20.644229 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8d66d765-ftb49_701c944e-55c1-4c6f-8f9e-164339efbe3d/kube-auth-proxy/0.log"