Sep 06 05:53:53.165468 ip-10-0-135-32 systemd[1]: Starting Kubernetes Kubelet... Sep 06 05:53:53.593263 ip-10-0-135-32 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 06 05:53:53.593263 ip-10-0-135-32 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 06 05:53:53.593263 ip-10-0-135-32 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 06 05:53:53.593263 ip-10-0-135-32 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 06 05:53:53.593263 ip-10-0-135-32 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 06 05:53:53.594458 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.594364 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 06 05:53:53.598738 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598714 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 05:53:53.598738 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598734 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 05:53:53.598738 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598740 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 05:53:53.598738 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598746 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598752 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598756 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598760 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598766 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598771 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598775 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598778 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598782 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598787 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598791 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598794 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598798 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598802 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598815 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598819 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598823 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598828 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598832 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 05:53:53.598989 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598836 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598839 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598843 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598847 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598851 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598855 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598858 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598862 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598867 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598871 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598875 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598879 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598884 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598888 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598892 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598898 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598902 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598906 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598910 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598917 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 05:53:53.599707 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598923 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598927 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598934 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598939 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598945 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598949 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598954 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598958 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598963 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598968 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598972 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598976 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598980 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598984 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598988 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598991 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.598996 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599000 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599004 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599008 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 06 05:53:53.600265 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599012 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599016 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599021 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599025 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599029 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599034 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599038 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599043 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599047 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599051 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599070 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599075 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599079 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599083 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599087 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599091 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599095 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599099 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599103 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599107 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 05:53:53.601084 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599111 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599117 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599121 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599125 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599129 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599133 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599137 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599141 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599145 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599149 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599153 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599157 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599161 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599166 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599190 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599194 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.599199 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601386 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601397 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601403 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 05:53:53.601949 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601408 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601413 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601418 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601422 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601426 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601430 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601435 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601439 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601443 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601447 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601451 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601457 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601464 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601469 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601476 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601480 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601484 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601489 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601493 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 05:53:53.602613 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601497 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601501 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601505 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601509 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601514 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601518 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601522 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601526 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601530 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601534 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601538 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601542 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601547 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601551 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601555 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601559 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601563 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601567 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601571 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 05:53:53.603102 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601578 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601583 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601588 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601592 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601596 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601600 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601604 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601608 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601614 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601619 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601623 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601627 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601631 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601635 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601639 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601643 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601647 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601653 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601657 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601661 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 05:53:53.603922 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601665 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601669 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601676 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601681 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601686 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601691 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601696 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601700 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601705 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601709 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601713 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601719 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601724 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601728 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601732 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601737 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601741 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601745 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601748 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 05:53:53.604813 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601752 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601757 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601761 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601766 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601770 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601774 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601778 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601782 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601786 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601790 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601794 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601798 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601803 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601807 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601811 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601815 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601818 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601822 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.601827 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603310 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 05:53:53.605570 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603323 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603328 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603333 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603337 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603342 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603346 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603350 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603354 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603359 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603363 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603367 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603372 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603376 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603380 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603385 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603391 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603396 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603400 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603404 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603408 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 05:53:53.606070 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603413 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603419 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603426 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603430 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603434 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603439 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603443 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603447 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603451 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603455 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603460 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603464 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603470 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603475 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603483 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603489 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603494 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603499 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 05:53:53.606695 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603504 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603508 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603512 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603516 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603520 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603525 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603529 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603533 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603537 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603541 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603546 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603550 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603553 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603557 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603562 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603567 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603571 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603575 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603578 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603583 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 05:53:53.607151 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603607 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603613 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603618 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603625 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603629 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603633 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603638 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603642 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603646 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603651 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603655 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603660 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603664 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603669 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603673 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603677 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603680 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603684 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603688 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603692 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 05:53:53.607742 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603697 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603701 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603705 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603709 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603714 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603718 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603722 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603726 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603730 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603734 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603738 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603742 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603746 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603750 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603753 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603758 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603761 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603765 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603769 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.603773 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 05:53:53.608364 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.603874 2578 flags.go:64] FLAG: --address="0.0.0.0" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.603929 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.603938 2578 flags.go:64] FLAG: --anonymous-auth="true" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.603944 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.603951 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.603956 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.603963 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.603970 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.603975 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.603980 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.603985 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.603991 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.603996 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604000 2578 flags.go:64] FLAG: --cgroup-root="" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604005 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604010 2578 flags.go:64] FLAG: --client-ca-file="" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604016 2578 flags.go:64] FLAG: --cloud-provider="external" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604021 2578 flags.go:64] FLAG: --cluster-dns="[]" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604027 2578 flags.go:64] FLAG: --cluster-domain="" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604032 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604036 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604041 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604047 2578 flags.go:64] FLAG: --container-log-max-files="5" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604053 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 06 05:53:53.608948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604058 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604063 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604069 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604074 2578 flags.go:64] FLAG: --contention-profiling="false" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604079 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604083 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604088 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604092 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604099 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604104 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604109 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604113 2578 flags.go:64] FLAG: --enable-load-reader="false" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604118 2578 flags.go:64] FLAG: --enable-server="true" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604122 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604129 2578 flags.go:64] FLAG: --event-burst="100" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604133 2578 flags.go:64] FLAG: --event-qps="50" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604138 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604144 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604149 2578 flags.go:64] FLAG: --eviction-hard="" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604155 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604159 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604164 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604169 2578 flags.go:64] FLAG: --eviction-soft="" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604195 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604200 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 06 05:53:53.609573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604205 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604210 2578 flags.go:64] FLAG: --experimental-mounter-path="" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604215 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604220 2578 flags.go:64] FLAG: --fail-swap-on="true" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604224 2578 flags.go:64] FLAG: --feature-gates="" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604234 2578 flags.go:64] FLAG: --file-check-frequency="20s" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604239 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604244 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604249 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604254 2578 flags.go:64] FLAG: --healthz-port="10248" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604259 2578 flags.go:64] FLAG: --help="false" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604263 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-135-32.ec2.internal" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604268 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604273 2578 flags.go:64] FLAG: --http-check-frequency="20s" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604278 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604284 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604290 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604295 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604300 2578 flags.go:64] FLAG: --image-service-endpoint="" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604304 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604309 2578 flags.go:64] FLAG: --kube-api-burst="100" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604313 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604318 2578 flags.go:64] FLAG: --kube-api-qps="50" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604323 2578 flags.go:64] FLAG: --kube-reserved="" Sep 06 05:53:53.610305 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604327 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604332 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604337 2578 flags.go:64] FLAG: --kubelet-cgroups="" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604341 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604345 2578 flags.go:64] FLAG: --lock-file="" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604350 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604354 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604359 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604368 2578 flags.go:64] FLAG: --log-json-split-stream="false" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604373 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604378 2578 flags.go:64] FLAG: --log-text-split-stream="false" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604382 2578 flags.go:64] FLAG: --logging-format="text" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604387 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604395 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604399 2578 flags.go:64] FLAG: --manifest-url="" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604404 2578 flags.go:64] FLAG: --manifest-url-header="" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604411 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604416 2578 flags.go:64] FLAG: --max-open-files="1000000" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604422 2578 flags.go:64] FLAG: --max-pods="110" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604427 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604432 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604436 2578 flags.go:64] FLAG: --memory-manager-policy="None" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604443 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604448 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604453 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 06 05:53:53.610901 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604458 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604469 2578 flags.go:64] FLAG: --node-status-max-images="50" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604474 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604479 2578 flags.go:64] FLAG: --oom-score-adj="-999" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604483 2578 flags.go:64] FLAG: --pod-cidr="" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604487 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604496 2578 flags.go:64] FLAG: --pod-manifest-path="" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604500 2578 flags.go:64] FLAG: --pod-max-pids="-1" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604505 2578 flags.go:64] FLAG: --pods-per-core="0" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604510 2578 flags.go:64] FLAG: --port="10250" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604514 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604519 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bd3cde35190eeb8e" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604524 2578 flags.go:64] FLAG: --qos-reserved="" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604529 2578 flags.go:64] FLAG: --read-only-port="10255" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604534 2578 flags.go:64] FLAG: --register-node="true" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604538 2578 flags.go:64] FLAG: --register-with-taints="" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604544 2578 flags.go:64] FLAG: --registry-burst="10" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604550 2578 flags.go:64] FLAG: --registry-qps="5" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604554 2578 flags.go:64] FLAG: --reserved-cpus="" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604559 2578 flags.go:64] FLAG: --reserved-memory="" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604567 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604573 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604578 2578 flags.go:64] FLAG: --rotate-certificates="false" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604583 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604587 2578 flags.go:64] FLAG: --runonce="false" Sep 06 05:53:53.611528 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604592 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604597 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604602 2578 flags.go:64] FLAG: --seccomp-default="false" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604607 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604611 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604618 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604623 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604628 2578 flags.go:64] FLAG: --storage-driver-password="root" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604633 2578 flags.go:64] FLAG: --storage-driver-secure="false" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604638 2578 flags.go:64] FLAG: --storage-driver-table="stats" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604642 2578 flags.go:64] FLAG: --storage-driver-user="root" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604647 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604651 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604656 2578 flags.go:64] FLAG: --system-cgroups="" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604661 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604669 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604673 2578 flags.go:64] FLAG: --tls-cert-file="" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604678 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604684 2578 flags.go:64] FLAG: --tls-min-version="" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604689 2578 flags.go:64] FLAG: --tls-private-key-file="" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604694 2578 flags.go:64] FLAG: --topology-manager-policy="none" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604698 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604703 2578 flags.go:64] FLAG: --topology-manager-scope="container" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604708 2578 flags.go:64] FLAG: --v="2" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604715 2578 flags.go:64] FLAG: --version="false" Sep 06 05:53:53.612139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604722 2578 flags.go:64] FLAG: --vmodule="" Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604728 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.604735 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.604924 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.604932 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.604937 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.604941 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.604946 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.604951 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.604955 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.604962 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.604969 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.604976 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.604982 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.604986 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.604990 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.604995 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.604999 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605003 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 05:53:53.612801 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605007 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605013 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605019 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605023 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605028 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605033 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605037 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605041 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605045 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605049 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605054 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605059 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605064 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605068 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605074 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605079 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605083 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605087 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605091 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605095 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 05:53:53.613355 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605099 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605104 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605108 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605112 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605116 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605124 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605129 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605133 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605137 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605142 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605146 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605150 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605154 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605159 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605163 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605167 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605188 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605192 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605197 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 05:53:53.613843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605201 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605205 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605210 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605214 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605218 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605222 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605226 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605230 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605237 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605241 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605245 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605249 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605253 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605257 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605262 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605265 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605269 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605273 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605279 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605283 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 05:53:53.614367 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605288 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605292 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605296 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605300 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605304 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605308 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605312 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605340 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605345 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605350 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605354 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605358 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605361 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605365 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605369 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605373 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605378 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605382 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605387 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605391 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 05:53:53.615118 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605396 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 05:53:53.615699 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605401 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 05:53:53.615699 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605405 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 05:53:53.615699 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.605409 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 05:53:53.615699 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.606362 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 06 05:53:53.616654 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.616634 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 06 05:53:53.616693 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.616655 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 06 05:53:53.617594 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.617584 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 06 05:53:53.617625 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.617595 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 06 05:53:53.617665 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617657 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 05:53:53.617665 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617665 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617670 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617676 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617680 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617684 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617688 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617690 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617693 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617696 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617699 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617701 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617704 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617707 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617710 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617712 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617715 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617717 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617720 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617722 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 05:53:53.617720 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617725 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617728 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617731 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617734 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617737 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617740 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617742 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617745 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617748 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617751 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617753 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617756 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617758 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617761 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617764 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617766 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617769 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617771 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617774 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617776 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 05:53:53.618194 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617779 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617781 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617783 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617786 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617788 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617791 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617793 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617796 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617799 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617801 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617804 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617807 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617810 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617812 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617815 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617817 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617820 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617822 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617825 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617827 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 05:53:53.618728 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617830 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617832 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617835 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617837 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617840 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617842 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617845 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617848 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617850 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617853 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617855 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617858 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617860 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617863 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617865 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617868 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617871 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617873 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617876 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 05:53:53.619308 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617880 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617884 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617887 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617890 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617893 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617896 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617898 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617901 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617904 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617907 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617910 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617913 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617916 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617918 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617921 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617923 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617926 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617928 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617931 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 05:53:53.619766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.617933 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 05:53:53.620252 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.617938 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 06 05:53:53.620252 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618084 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 06 05:53:53.620252 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618090 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 06 05:53:53.620252 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618093 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 06 05:53:53.620252 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618096 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 06 05:53:53.620252 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618099 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 06 05:53:53.620252 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618101 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 06 05:53:53.620252 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618106 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 06 05:53:53.620252 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618110 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 06 05:53:53.620252 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618114 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 06 05:53:53.620252 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618116 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 06 05:53:53.620252 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618119 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 06 05:53:53.620252 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618122 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 06 05:53:53.620252 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618125 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 06 05:53:53.620252 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618128 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618131 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618134 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618136 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618139 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618141 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618144 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618146 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618149 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618151 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618153 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618156 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618158 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618161 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618165 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618168 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618186 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618190 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618193 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 06 05:53:53.620650 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618195 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618198 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618201 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618203 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618206 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618208 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618211 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618213 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618217 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618219 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618221 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618224 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618227 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618230 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618233 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618235 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618238 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618240 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618243 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618245 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 06 05:53:53.621101 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618249 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618251 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618254 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618256 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618258 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618261 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618263 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618266 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618268 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618271 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618273 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618275 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618278 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618280 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618282 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618285 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618287 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618289 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618292 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618296 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 06 05:53:53.621605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618299 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618302 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618304 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618307 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618310 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618313 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618323 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618325 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618328 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618330 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618333 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618336 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618338 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618341 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618344 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618346 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618349 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618351 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618354 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618356 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 06 05:53:53.622089 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618358 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 06 05:53:53.622583 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618361 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 06 05:53:53.622583 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618364 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 06 05:53:53.622583 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618366 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 06 05:53:53.622583 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618368 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 06 05:53:53.622583 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618371 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 06 05:53:53.622583 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:53.618373 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 06 05:53:53.622583 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.618377 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 06 05:53:53.622583 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.618509 2578 server.go:962] "Client rotation is on, will bootstrap in background" Sep 06 05:53:53.622583 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.621282 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 06 05:53:53.622583 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.622319 2578 server.go:1019] "Starting client certificate rotation" Sep 06 05:53:53.622583 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.622425 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 06 05:53:53.622583 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.622459 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 06 05:53:53.625388 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.625371 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 06 05:53:53.626579 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.626562 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 06 05:53:53.637523 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.637503 2578 log.go:25] "Validated CRI v1 runtime API" Sep 06 05:53:53.641018 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.640993 2578 log.go:25] "Validated CRI v1 image API" Sep 06 05:53:53.641944 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.641928 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 06 05:53:53.646879 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.646861 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 98d3bea2-6dfa-47a7-845f-f361301b6220:/dev/nvme0n1p3 fd7f05b1-203e-41ea-9041-01ef346aa3cb:/dev/nvme0n1p4] Sep 06 05:53:53.646948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.646879 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 06 05:53:53.653296 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.653118 2578 manager.go:216] Machine: {Timestamp:2026-09-06 05:53:53.651683659 +0000 UTC m=+0.380742423 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3126220 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2988e559e6a125c76ff56476fe18b9 SystemUUID:ec2988e5-59e6-a125-c76f-f56476fe18b9 BootID:c285a83c-5871-4a96-aa46-6128d5c5be21 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1e:fe:1f:95:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1e:fe:1f:95:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:28:a8:72:11:fa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 06 05:53:53.653296 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.653288 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 06 05:53:53.653452 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.653410 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 06 05:53:53.657070 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.657041 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 06 05:53:53.657264 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.657073 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-32.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 06 05:53:53.658690 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.658677 2578 topology_manager.go:138] "Creating topology manager with none policy" Sep 06 05:53:53.658723 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.658696 2578 container_manager_linux.go:309] "Creating device plugin manager" Sep 06 05:53:53.658723 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.658704 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 06 05:53:53.659560 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.659547 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 06 05:53:53.660797 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.660785 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 06 05:53:53.660999 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.660981 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 06 05:53:53.666325 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.666310 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 06 05:53:53.666452 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.666442 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 06 05:53:53.666481 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.666474 2578 kubelet.go:486] "Attempting to sync node with API server" Sep 06 05:53:53.666517 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.666484 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 06 05:53:53.666517 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.666500 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 06 05:53:53.666517 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.666509 2578 kubelet.go:398] "Adding apiserver pod source" Sep 06 05:53:53.666517 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.666518 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 06 05:53:53.667069 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.667046 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 06 05:53:53.667103 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.667080 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 06 05:53:53.669701 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.669679 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 06 05:53:53.673044 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.673029 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 06 05:53:53.673108 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.673060 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 06 05:53:53.673894 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.673881 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 06 05:53:53.673943 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.673900 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 06 05:53:53.673943 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.673907 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 06 05:53:53.673943 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.673920 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 06 05:53:53.673943 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.673932 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 06 05:53:53.673943 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.673941 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 06 05:53:53.674079 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.673954 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 06 05:53:53.674079 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.673960 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 06 05:53:53.674079 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.673967 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 06 05:53:53.674079 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.673973 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 06 05:53:53.674079 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.673982 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 06 05:53:53.674079 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.673988 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 06 05:53:53.674874 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.674863 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 06 05:53:53.674903 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.674875 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 06 05:53:53.675854 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:53.675828 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-32.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 06 05:53:53.675900 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:53.675829 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 06 05:53:53.678190 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.678151 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mcbkj" Sep 06 05:53:53.679153 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.679138 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-32.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 06 05:53:53.680121 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.680103 2578 server.go:1268] "Started kubelet" Sep 06 05:53:53.680232 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.680147 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 06 05:53:53.680232 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.680212 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 06 05:53:53.680324 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.680256 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 06 05:53:53.680995 ip-10-0-135-32 systemd[1]: Started Kubernetes Kubelet. Sep 06 05:53:53.681832 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.681815 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 06 05:53:53.682532 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.682518 2578 server.go:310] "Adding debug handlers to kubelet server" Sep 06 05:53:53.684415 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:53.684394 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 06 05:53:53.686339 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:53.685162 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-32.ec2.internal.18d2a58e8c748880 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-32.ec2.internal,UID:ip-10-0-135-32.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-32.ec2.internal,},FirstTimestamp:2026-09-06 05:53:53.679894656 +0000 UTC m=+0.408953408,LastTimestamp:2026-09-06 05:53:53.679894656 +0000 UTC m=+0.408953408,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-32.ec2.internal,}" Sep 06 05:53:53.688832 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.688816 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 06 05:53:53.689504 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.689486 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 06 05:53:53.689813 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.689779 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mcbkj" Sep 06 05:53:53.689813 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:53.689780 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Sep 06 05:53:53.690422 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.690404 2578 factory.go:55] Registering systemd factory Sep 06 05:53:53.690422 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.690418 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 06 05:53:53.690523 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.690427 2578 factory.go:223] Registration of the systemd container factory successfully Sep 06 05:53:53.690629 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.690489 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 06 05:53:53.690629 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.690595 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 06 05:53:53.690731 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.690652 2578 factory.go:153] Registering CRI-O factory Sep 06 05:53:53.690731 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.690668 2578 factory.go:223] Registration of the crio container factory successfully Sep 06 05:53:53.690731 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.690699 2578 reconstruct.go:97] "Volume reconstruction finished" Sep 06 05:53:53.690731 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.690706 2578 reconciler.go:29] "Reconciler: start to sync state" Sep 06 05:53:53.690731 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.690720 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 06 05:53:53.690941 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.690739 2578 factory.go:103] Registering Raw factory Sep 06 05:53:53.690941 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.690754 2578 manager.go:1196] Started watching for new ooms in manager Sep 06 05:53:53.691164 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.691147 2578 manager.go:349] Starting recovery of all containers Sep 06 05:53:53.713427 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.713398 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 05:53:53.713762 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.713704 2578 manager.go:354] Recovery completed Sep 06 05:53:53.718084 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.718065 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 05:53:53.719270 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:53.719251 2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-32.ec2.internal\" not found" node="ip-10-0-135-32.ec2.internal" Sep 06 05:53:53.720025 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.720005 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientMemory" Sep 06 05:53:53.720117 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.720037 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 05:53:53.720117 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.720051 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientPID" Sep 06 05:53:53.720699 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.720686 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 06 05:53:53.720699 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.720696 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 05:53:53.720797 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.720697 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 06 05:53:53.720797 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.720736 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 06 05:53:53.721430 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.721408 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientMemory" Sep 06 05:53:53.721503 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.721443 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 05:53:53.721503 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.721457 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientPID" Sep 06 05:53:53.722805 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.722792 2578 policy_none.go:49] "None policy: Start" Sep 06 05:53:53.722851 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.722809 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 06 05:53:53.722851 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.722819 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 06 05:53:53.724835 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.724822 2578 policy_none.go:47] "Start" Sep 06 05:53:53.780649 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.760736 2578 manager.go:337] "Starting Device Plugin manager" Sep 06 05:53:53.780649 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:53.760904 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 06 05:53:53.780649 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.760915 2578 server.go:87] "Starting device plugin registration server" Sep 06 05:53:53.780649 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.761143 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 06 05:53:53.780649 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.761153 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 06 05:53:53.780649 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.761253 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 06 05:53:53.780649 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.761337 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 06 05:53:53.780649 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.761345 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 06 05:53:53.780649 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:53.761921 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 06 05:53:53.780649 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:53.761977 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-32.ec2.internal\" not found" Sep 06 05:53:53.805793 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.805748 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 06 05:53:53.807112 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.807085 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 06 05:53:53.807256 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.807163 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 06 05:53:53.807256 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.807214 2578 kubelet.go:2448] "Starting kubelet main sync loop" Sep 06 05:53:53.807356 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:53.807263 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 06 05:53:53.810252 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.810232 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 05:53:53.861681 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.861586 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 05:53:53.862699 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.862680 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientMemory" Sep 06 05:53:53.862791 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.862712 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 05:53:53.862791 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.862728 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientPID" Sep 06 05:53:53.862791 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.862760 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-32.ec2.internal" Sep 06 05:53:53.869621 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.869588 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-32.ec2.internal" Sep 06 05:53:53.869621 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:53.869620 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-32.ec2.internal\": node \"ip-10-0-135-32.ec2.internal\" not found" Sep 06 05:53:53.890431 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:53.890397 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Sep 06 05:53:53.907907 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.907878 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal"] Sep 06 05:53:53.907999 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.907960 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 05:53:53.909760 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.909736 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientMemory" Sep 06 05:53:53.909850 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.909772 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 05:53:53.909850 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.909782 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientPID" Sep 06 05:53:53.912108 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.912096 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 05:53:53.912298 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.912283 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Sep 06 05:53:53.912338 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.912322 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 05:53:53.913035 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.913015 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientMemory" Sep 06 05:53:53.913137 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.913045 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientMemory" Sep 06 05:53:53.913137 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.913052 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 05:53:53.913137 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.913072 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientPID" Sep 06 05:53:53.913137 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.913089 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 05:53:53.913137 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.913101 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientPID" Sep 06 05:53:53.915514 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.915498 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" Sep 06 05:53:53.915587 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.915522 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 06 05:53:53.916395 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.916376 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientMemory" Sep 06 05:53:53.916496 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.916403 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasNoDiskPressure" Sep 06 05:53:53.916496 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:53.916413 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeHasSufficientPID" Sep 06 05:53:53.939991 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:53.939969 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-32.ec2.internal\" not found" node="ip-10-0-135-32.ec2.internal" Sep 06 05:53:53.944529 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:53.944511 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-32.ec2.internal\" not found" node="ip-10-0-135-32.ec2.internal" Sep 06 05:53:53.991387 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:53.991358 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Sep 06 05:53:54.091904 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:54.091870 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Sep 06 05:53:54.092054 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.091917 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b74d93d1db0c298014dfe267b0549d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal\" (UID: \"4b74d93d1db0c298014dfe267b0549d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Sep 06 05:53:54.092054 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.091944 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b74d93d1db0c298014dfe267b0549d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal\" (UID: \"4b74d93d1db0c298014dfe267b0549d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Sep 06 05:53:54.092054 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.091961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd1de5c49e7422a298303b9a2c5c9a84-config\") pod \"kube-apiserver-proxy-ip-10-0-135-32.ec2.internal\" (UID: \"bd1de5c49e7422a298303b9a2c5c9a84\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" Sep 06 05:53:54.192626 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:54.192524 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Sep 06 05:53:54.192626 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.192605 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b74d93d1db0c298014dfe267b0549d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal\" (UID: \"4b74d93d1db0c298014dfe267b0549d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Sep 06 05:53:54.192804 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.192641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b74d93d1db0c298014dfe267b0549d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal\" (UID: \"4b74d93d1db0c298014dfe267b0549d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Sep 06 05:53:54.192804 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.192673 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd1de5c49e7422a298303b9a2c5c9a84-config\") pod \"kube-apiserver-proxy-ip-10-0-135-32.ec2.internal\" (UID: \"bd1de5c49e7422a298303b9a2c5c9a84\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" Sep 06 05:53:54.192804 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.192696 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b74d93d1db0c298014dfe267b0549d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal\" (UID: \"4b74d93d1db0c298014dfe267b0549d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Sep 06 05:53:54.192804 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.192713 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b74d93d1db0c298014dfe267b0549d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal\" (UID: \"4b74d93d1db0c298014dfe267b0549d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Sep 06 05:53:54.192804 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.192710 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd1de5c49e7422a298303b9a2c5c9a84-config\") pod \"kube-apiserver-proxy-ip-10-0-135-32.ec2.internal\" (UID: \"bd1de5c49e7422a298303b9a2c5c9a84\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" Sep 06 05:53:54.241767 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.241722 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Sep 06 05:53:54.247499 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.247235 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" Sep 06 05:53:54.293046 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:54.293013 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Sep 06 05:53:54.393483 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:54.393453 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Sep 06 05:53:54.493921 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:54.493831 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Sep 06 05:53:54.594318 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:54.594286 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Sep 06 05:53:54.622499 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.622470 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 06 05:53:54.622657 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.622636 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 06 05:53:54.622700 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.622680 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 06 05:53:54.694369 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:54.694336 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Sep 06 05:53:54.711086 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.711038 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-05 05:48:55 +0000 UTC" deadline="2028-06-06 18:35:58.027576408 +0000 UTC" Sep 06 05:53:54.711086 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.711079 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15348h42m3.316500846s" Sep 06 05:53:54.795137 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:54.795106 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-32.ec2.internal\" not found" Sep 06 05:53:54.828553 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.828525 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 05:53:54.890188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.890144 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" Sep 06 05:53:54.900599 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.900569 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 06 05:53:54.902478 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.902463 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" Sep 06 05:53:54.908188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.908157 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 06 05:53:54.924387 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:54.924349 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b74d93d1db0c298014dfe267b0549d7.slice/crio-d5557fb8a61bb98fec0d62eb6e94818a2b2a462f33bfec583e068d947f9af61c WatchSource:0}: Error finding container d5557fb8a61bb98fec0d62eb6e94818a2b2a462f33bfec583e068d947f9af61c: Status 404 returned error can't find the container with id d5557fb8a61bb98fec0d62eb6e94818a2b2a462f33bfec583e068d947f9af61c Sep 06 05:53:54.924605 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:54.924585 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1de5c49e7422a298303b9a2c5c9a84.slice/crio-1777cf42267c1179bdacd3e89c933f1b9c85f7fe54e5e17b2ed626df53d68122 WatchSource:0}: Error finding container 1777cf42267c1179bdacd3e89c933f1b9c85f7fe54e5e17b2ed626df53d68122: Status 404 returned error can't find the container with id 1777cf42267c1179bdacd3e89c933f1b9c85f7fe54e5e17b2ed626df53d68122 Sep 06 05:53:54.929511 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:54.929495 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 05:53:55.028284 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:55.028202 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 05:53:56.998258 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:56.998229 2578 apiserver.go:52] "Watching apiserver" Sep 06 05:53:57.003393 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.003359 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 06 05:53:57.005058 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.005025 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-4b45g","openshift-multus/multus-dm4ww","kube-system/global-pull-secret-syncer-c66ws","openshift-cluster-node-tuning-operator/tuned-g625l","openshift-multus/network-metrics-daemon-w74pq","openshift-network-diagnostics/network-check-target-f87xg","openshift-network-operator/iptables-alerter-dknd9","openshift-ovn-kubernetes/ovnkube-node-dwh56","kube-system/konnectivity-agent-9c7xb","kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs","openshift-dns/node-resolver-qcvmz","openshift-image-registry/node-ca-6fw4z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal"] Sep 06 05:53:57.009718 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.008036 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dknd9" Sep 06 05:53:57.012275 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.012249 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v9d7h\"" Sep 06 05:53:57.012482 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.012467 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 06 05:53:57.012645 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.012631 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 06 05:53:57.012815 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.012800 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 06 05:53:57.015927 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.015150 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6fw4z" Sep 06 05:53:57.017937 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.016854 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 06 05:53:57.017937 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.017232 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k5xwn\"" Sep 06 05:53:57.017937 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.017369 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 06 05:53:57.017937 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.017530 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 06 05:53:57.017937 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.017633 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:53:57.017937 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.017699 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:53:57.020112 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.020004 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.021805 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.021787 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:53:57.021927 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.021852 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:53:57.021995 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.021974 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 06 05:53:57.022890 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.022872 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wtxjc\"" Sep 06 05:53:57.022890 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.022887 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 06 05:53:57.023151 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.023133 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 06 05:53:57.023971 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.023955 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:53:57.024058 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.024015 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:53:57.026678 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.026613 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9c7xb" Sep 06 05:53:57.028496 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.028427 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 06 05:53:57.028738 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.028724 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2z66f\"" Sep 06 05:53:57.028968 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.028951 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 06 05:53:57.029046 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.028988 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.031374 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031350 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-lib-modules\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.031459 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031381 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-sysconfig\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.031459 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-host\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.031459 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031434 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-tuned\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.031604 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-tmp\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.031604 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031478 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jkfd\" (UniqueName: \"kubernetes.io/projected/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-kube-api-access-8jkfd\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.031604 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:53:57.031604 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031517 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkj5g\" (UniqueName: \"kubernetes.io/projected/f36acd55-4ce4-4bdb-9518-cca261651a19-kube-api-access-vkj5g\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:53:57.031604 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031537 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/62afc7f8-8ed3-4dbe-95db-e585f70422f8-iptables-alerter-script\") pod \"iptables-alerter-dknd9\" (UID: \"62afc7f8-8ed3-4dbe-95db-e585f70422f8\") " pod="openshift-network-operator/iptables-alerter-dknd9" Sep 06 05:53:57.031604 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bddf45b0-9199-4a9d-aee3-ca253d8a9c56-serviceca\") pod \"node-ca-6fw4z\" (UID: \"bddf45b0-9199-4a9d-aee3-ca253d8a9c56\") " pod="openshift-image-registry/node-ca-6fw4z" Sep 06 05:53:57.031604 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031574 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-sysctl-conf\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.031604 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-run\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.031604 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-var-lib-kubelet\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.032024 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031626 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62afc7f8-8ed3-4dbe-95db-e585f70422f8-host-slash\") pod \"iptables-alerter-dknd9\" (UID: \"62afc7f8-8ed3-4dbe-95db-e585f70422f8\") " pod="openshift-network-operator/iptables-alerter-dknd9" Sep 06 05:53:57.032024 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bddf45b0-9199-4a9d-aee3-ca253d8a9c56-host\") pod \"node-ca-6fw4z\" (UID: \"bddf45b0-9199-4a9d-aee3-ca253d8a9c56\") " pod="openshift-image-registry/node-ca-6fw4z" Sep 06 05:53:57.032024 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031661 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m8ts\" (UniqueName: \"kubernetes.io/projected/bddf45b0-9199-4a9d-aee3-ca253d8a9c56-kube-api-access-5m8ts\") pod \"node-ca-6fw4z\" (UID: \"bddf45b0-9199-4a9d-aee3-ca253d8a9c56\") " pod="openshift-image-registry/node-ca-6fw4z" Sep 06 05:53:57.032024 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031677 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/115882c1-6556-4feb-adc9-a5ddf5f36ba2-kubelet-config\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:53:57.032024 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031701 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/115882c1-6556-4feb-adc9-a5ddf5f36ba2-dbus\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:53:57.032024 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-modprobe-d\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.032024 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031760 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-systemd\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.032024 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031790 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpvdd\" (UniqueName: \"kubernetes.io/projected/62afc7f8-8ed3-4dbe-95db-e585f70422f8-kube-api-access-hpvdd\") pod \"iptables-alerter-dknd9\" (UID: \"62afc7f8-8ed3-4dbe-95db-e585f70422f8\") " pod="openshift-network-operator/iptables-alerter-dknd9" Sep 06 05:53:57.032024 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031819 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:53:57.032024 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031853 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-kubernetes\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.032024 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031882 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-sysctl-d\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.032024 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.031939 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-sys\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.036515 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.036483 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.037932 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.037474 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 06 05:53:57.037932 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.037572 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sqcw2\"" Sep 06 05:53:57.037932 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.037840 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 06 05:53:57.037932 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.037865 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 06 05:53:57.037932 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.037876 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 06 05:53:57.038218 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.037935 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 06 05:53:57.038259 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.038235 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bjcz8\"" Sep 06 05:53:57.038301 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.038286 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 06 05:53:57.038467 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.038445 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 06 05:53:57.038736 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.038720 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.040427 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.040412 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 06 05:53:57.040792 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.040775 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 06 05:53:57.040952 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.040939 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 06 05:53:57.041002 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.040947 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r2fj7\"" Sep 06 05:53:57.041048 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.041035 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 06 05:53:57.041150 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.041135 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 06 05:53:57.041476 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.041460 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 06 05:53:57.042457 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.042438 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.044452 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.044432 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 06 05:53:57.044533 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.044506 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qcvmz" Sep 06 05:53:57.044679 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.044663 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 06 05:53:57.045040 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.045021 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9qfq8\"" Sep 06 05:53:57.045125 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.045060 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 06 05:53:57.046448 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.046375 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 06 05:53:57.046760 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.046729 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 06 05:53:57.047009 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.046955 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z45b4\"" Sep 06 05:53:57.053175 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.053141 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qrqjx" Sep 06 05:53:57.060410 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.060364 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qrqjx" Sep 06 05:53:57.123201 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.123169 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 06 05:53:57.132588 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-tuned\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.132785 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132599 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkfd\" (UniqueName: \"kubernetes.io/projected/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-kube-api-access-8jkfd\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.132785 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-registration-dir\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.132785 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132658 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-multus-socket-dir-parent\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.132785 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e97d5427-3253-45fd-8a75-b0f4a8dd0452-hosts-file\") pod \"node-resolver-qcvmz\" (UID: \"e97d5427-3253-45fd-8a75-b0f4a8dd0452\") " pod="openshift-dns/node-resolver-qcvmz" Sep 06 05:53:57.132785 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/62afc7f8-8ed3-4dbe-95db-e585f70422f8-iptables-alerter-script\") pod \"iptables-alerter-dknd9\" (UID: \"62afc7f8-8ed3-4dbe-95db-e585f70422f8\") " pod="openshift-network-operator/iptables-alerter-dknd9" Sep 06 05:53:57.132785 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132737 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-log-socket\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.132785 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.133188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132797 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-run-multus-certs\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.133188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-sys\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.133188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132852 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/115882c1-6556-4feb-adc9-a5ddf5f36ba2-kubelet-config\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:53:57.133188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132876 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-systemd-units\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.133188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132916 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czz56\" (UniqueName: \"kubernetes.io/projected/210471f4-8d5c-41ae-afa1-6deccca7846e-kube-api-access-czz56\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.133188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-etc-openvswitch\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.133188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132972 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.133188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.132994 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-system-cni-dir\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.133188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133018 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a837857-cbb6-411d-8d9c-1af494b92d13-cni-binary-copy\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.133188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133041 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97d5427-3253-45fd-8a75-b0f4a8dd0452-tmp-dir\") pod \"node-resolver-qcvmz\" (UID: \"e97d5427-3253-45fd-8a75-b0f4a8dd0452\") " pod="openshift-dns/node-resolver-qcvmz" Sep 06 05:53:57.133188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133067 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-var-lib-kubelet\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.133188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/faf01480-6d08-4434-822f-28b4815084d2-agent-certs\") pod \"konnectivity-agent-9c7xb\" (UID: \"faf01480-6d08-4434-822f-28b4815084d2\") " pod="kube-system/konnectivity-agent-9c7xb" Sep 06 05:53:57.133188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/557fb632-fd3a-4b82-b25a-ba2e25072337-cnibin\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.133188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133146 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-run\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.133188 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/557fb632-fd3a-4b82-b25a-ba2e25072337-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133209 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62afc7f8-8ed3-4dbe-95db-e585f70422f8-host-slash\") pod \"iptables-alerter-dknd9\" (UID: \"62afc7f8-8ed3-4dbe-95db-e585f70422f8\") " pod="openshift-network-operator/iptables-alerter-dknd9" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-run-systemd\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-run-openvswitch\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133280 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-sys-fs\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133303 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmnfh\" (UniqueName: \"kubernetes.io/projected/e97d5427-3253-45fd-8a75-b0f4a8dd0452-kube-api-access-hmnfh\") pod \"node-resolver-qcvmz\" (UID: \"e97d5427-3253-45fd-8a75-b0f4a8dd0452\") " pod="openshift-dns/node-resolver-qcvmz" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133325 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/faf01480-6d08-4434-822f-28b4815084d2-konnectivity-ca\") pod \"konnectivity-agent-9c7xb\" (UID: \"faf01480-6d08-4434-822f-28b4815084d2\") " pod="kube-system/konnectivity-agent-9c7xb" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133350 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/557fb632-fd3a-4b82-b25a-ba2e25072337-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-sysctl-d\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-slash\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133418 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-run-ovn\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133441 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4532703-7257-43ee-b5bd-fc18dd83a77e-env-overrides\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133468 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-var-lib-cni-multus\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-sysctl-conf\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-var-lib-kubelet\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133541 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-kubelet\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133766 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-node-log\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.133811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133810 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4532703-7257-43ee-b5bd-fc18dd83a77e-ovn-node-metrics-cert\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.134636 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133858 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-multus-cni-dir\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.134636 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133891 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-run-netns\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.134636 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133939 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-hostroot\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.134636 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.133982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-lib-modules\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.134636 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.134185 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/115882c1-6556-4feb-adc9-a5ddf5f36ba2-dbus\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:53:57.134636 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.134225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-modprobe-d\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.134636 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.134262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-systemd\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.134636 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.134295 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmplp\" (UniqueName: \"kubernetes.io/projected/9a837857-cbb6-411d-8d9c-1af494b92d13-kube-api-access-dmplp\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.134636 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.134329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/557fb632-fd3a-4b82-b25a-ba2e25072337-system-cni-dir\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.134636 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.134357 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/557fb632-fd3a-4b82-b25a-ba2e25072337-cni-binary-copy\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.134636 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.134392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpvdd\" (UniqueName: \"kubernetes.io/projected/62afc7f8-8ed3-4dbe-95db-e585f70422f8-kube-api-access-hpvdd\") pod \"iptables-alerter-dknd9\" (UID: \"62afc7f8-8ed3-4dbe-95db-e585f70422f8\") " pod="openshift-network-operator/iptables-alerter-dknd9" Sep 06 05:53:57.134636 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.134598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-run-ovn-kubernetes\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.134636 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.134632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-device-dir\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.135348 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.134661 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-run-k8s-cni-cncf-io\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.135348 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.134697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-host\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.135348 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.134728 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-tmp\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.135348 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.134762 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-cni-bin\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.138078 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138043 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-cnibin\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.138214 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/557fb632-fd3a-4b82-b25a-ba2e25072337-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.138214 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:53:57.138214 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkj5g\" (UniqueName: \"kubernetes.io/projected/f36acd55-4ce4-4bdb-9518-cca261651a19-kube-api-access-vkj5g\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:53:57.138378 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138221 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bddf45b0-9199-4a9d-aee3-ca253d8a9c56-serviceca\") pod \"node-ca-6fw4z\" (UID: \"bddf45b0-9199-4a9d-aee3-ca253d8a9c56\") " pod="openshift-image-registry/node-ca-6fw4z" Sep 06 05:53:57.138378 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-cni-netd\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.138378 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4532703-7257-43ee-b5bd-fc18dd83a77e-ovnkube-config\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.138378 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138323 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4532703-7257-43ee-b5bd-fc18dd83a77e-ovnkube-script-lib\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.138378 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-socket-dir\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.138644 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-multus-conf-dir\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.138644 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bddf45b0-9199-4a9d-aee3-ca253d8a9c56-host\") pod \"node-ca-6fw4z\" (UID: \"bddf45b0-9199-4a9d-aee3-ca253d8a9c56\") " pod="openshift-image-registry/node-ca-6fw4z" Sep 06 05:53:57.138644 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138468 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m8ts\" (UniqueName: \"kubernetes.io/projected/bddf45b0-9199-4a9d-aee3-ca253d8a9c56-kube-api-access-5m8ts\") pod \"node-ca-6fw4z\" (UID: \"bddf45b0-9199-4a9d-aee3-ca253d8a9c56\") " pod="openshift-image-registry/node-ca-6fw4z" Sep 06 05:53:57.138644 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-etc-selinux\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.138644 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a837857-cbb6-411d-8d9c-1af494b92d13-multus-daemon-config\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.138644 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138591 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bn8x\" (UniqueName: \"kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x\") pod \"network-check-target-f87xg\" (UID: \"a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40\") " pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:53:57.138644 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:53:57.138997 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138666 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-kubernetes\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.138997 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138695 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-run-netns\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.138997 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138728 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-var-lib-openvswitch\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.138997 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138759 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nm6l\" (UniqueName: \"kubernetes.io/projected/e4532703-7257-43ee-b5bd-fc18dd83a77e-kube-api-access-8nm6l\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.138997 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138790 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-var-lib-cni-bin\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.138997 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138817 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-etc-kubernetes\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.138997 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6cn8\" (UniqueName: \"kubernetes.io/projected/557fb632-fd3a-4b82-b25a-ba2e25072337-kube-api-access-v6cn8\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.138997 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.138885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-sysconfig\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.139378 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.139023 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-sysconfig\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.139378 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.139203 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:53:57.139378 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.139294 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs podName:f36acd55-4ce4-4bdb-9518-cca261651a19 nodeName:}" failed. No retries permitted until 2026-09-06 05:53:57.639261025 +0000 UTC m=+3.037591495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs") pod "network-metrics-daemon-w74pq" (UID: "f36acd55-4ce4-4bdb-9518-cca261651a19") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:53:57.140309 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.140282 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bddf45b0-9199-4a9d-aee3-ca253d8a9c56-serviceca\") pod \"node-ca-6fw4z\" (UID: \"bddf45b0-9199-4a9d-aee3-ca253d8a9c56\") " pod="openshift-image-registry/node-ca-6fw4z" Sep 06 05:53:57.140478 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.140459 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bddf45b0-9199-4a9d-aee3-ca253d8a9c56-host\") pod \"node-ca-6fw4z\" (UID: \"bddf45b0-9199-4a9d-aee3-ca253d8a9c56\") " pod="openshift-image-registry/node-ca-6fw4z" Sep 06 05:53:57.140866 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.140845 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 05:53:57.141003 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.140928 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret podName:115882c1-6556-4feb-adc9-a5ddf5f36ba2 nodeName:}" failed. No retries permitted until 2026-09-06 05:53:57.640894234 +0000 UTC m=+3.039224684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret") pod "global-pull-secret-syncer-c66ws" (UID: "115882c1-6556-4feb-adc9-a5ddf5f36ba2") : object "kube-system"/"original-pull-secret" not registered Sep 06 05:53:57.141003 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.140994 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-kubernetes\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.143202 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.143167 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 06 05:53:57.144117 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.143761 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-sysctl-d\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.144117 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.143991 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-sysctl-conf\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.144117 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.144066 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-var-lib-kubelet\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.144328 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.144274 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-lib-modules\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.144440 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.144383 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-systemd\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.144440 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.144426 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/115882c1-6556-4feb-adc9-a5ddf5f36ba2-dbus\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:53:57.144538 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.144506 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-modprobe-d\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.144754 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.144696 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-host\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.145678 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.145640 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/62afc7f8-8ed3-4dbe-95db-e585f70422f8-iptables-alerter-script\") pod \"iptables-alerter-dknd9\" (UID: \"62afc7f8-8ed3-4dbe-95db-e585f70422f8\") " pod="openshift-network-operator/iptables-alerter-dknd9" Sep 06 05:53:57.145937 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.145917 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-sys\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.146028 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.146008 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/115882c1-6556-4feb-adc9-a5ddf5f36ba2-kubelet-config\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:53:57.146155 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.146137 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-run\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.146368 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.146350 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62afc7f8-8ed3-4dbe-95db-e585f70422f8-host-slash\") pod \"iptables-alerter-dknd9\" (UID: \"62afc7f8-8ed3-4dbe-95db-e585f70422f8\") " pod="openshift-network-operator/iptables-alerter-dknd9" Sep 06 05:53:57.148413 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.148387 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m8ts\" (UniqueName: \"kubernetes.io/projected/bddf45b0-9199-4a9d-aee3-ca253d8a9c56-kube-api-access-5m8ts\") pod \"node-ca-6fw4z\" (UID: \"bddf45b0-9199-4a9d-aee3-ca253d8a9c56\") " pod="openshift-image-registry/node-ca-6fw4z" Sep 06 05:53:57.149082 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.149010 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-etc-tuned\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.149082 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.149031 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkj5g\" (UniqueName: \"kubernetes.io/projected/f36acd55-4ce4-4bdb-9518-cca261651a19-kube-api-access-vkj5g\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:53:57.151307 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.151244 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-tmp\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.155670 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.155645 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkfd\" (UniqueName: \"kubernetes.io/projected/f0af07e4-9ff9-462f-a6cc-0d4e9a37de97-kube-api-access-8jkfd\") pod \"tuned-g625l\" (UID: \"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97\") " pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.158490 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.158441 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" event={"ID":"bd1de5c49e7422a298303b9a2c5c9a84","Type":"ContainerStarted","Data":"1777cf42267c1179bdacd3e89c933f1b9c85f7fe54e5e17b2ed626df53d68122"} Sep 06 05:53:57.160401 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.160380 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpvdd\" (UniqueName: \"kubernetes.io/projected/62afc7f8-8ed3-4dbe-95db-e585f70422f8-kube-api-access-hpvdd\") pod \"iptables-alerter-dknd9\" (UID: \"62afc7f8-8ed3-4dbe-95db-e585f70422f8\") " pod="openshift-network-operator/iptables-alerter-dknd9" Sep 06 05:53:57.160688 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.160658 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" event={"ID":"4b74d93d1db0c298014dfe267b0549d7","Type":"ContainerStarted","Data":"d5557fb8a61bb98fec0d62eb6e94818a2b2a462f33bfec583e068d947f9af61c"} Sep 06 05:53:57.233400 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.233364 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 05:53:57.240043 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240010 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-cni-bin\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.240212 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-cnibin\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.240212 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240082 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/557fb632-fd3a-4b82-b25a-ba2e25072337-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.240212 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-cni-netd\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.240212 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240134 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-cni-bin\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.240212 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4532703-7257-43ee-b5bd-fc18dd83a77e-ovnkube-config\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.240212 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240157 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-cnibin\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.240212 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4532703-7257-43ee-b5bd-fc18dd83a77e-ovnkube-script-lib\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.240212 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240213 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-socket-dir\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.240212 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240214 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-cni-netd\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-multus-conf-dir\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240268 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-etc-selinux\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240287 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a837857-cbb6-411d-8d9c-1af494b92d13-multus-daemon-config\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bn8x\" (UniqueName: \"kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x\") pod \"network-check-target-f87xg\" (UID: \"a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40\") " pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240344 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-run-netns\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-var-lib-openvswitch\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nm6l\" (UniqueName: \"kubernetes.io/projected/e4532703-7257-43ee-b5bd-fc18dd83a77e-kube-api-access-8nm6l\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240388 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-var-lib-cni-bin\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-etc-kubernetes\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240429 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6cn8\" (UniqueName: \"kubernetes.io/projected/557fb632-fd3a-4b82-b25a-ba2e25072337-kube-api-access-v6cn8\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-registration-dir\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-multus-socket-dir-parent\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240498 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e97d5427-3253-45fd-8a75-b0f4a8dd0452-hosts-file\") pod \"node-resolver-qcvmz\" (UID: \"e97d5427-3253-45fd-8a75-b0f4a8dd0452\") " pod="openshift-dns/node-resolver-qcvmz" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240514 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-log-socket\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240539 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-run-multus-certs\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.240633 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-systemd-units\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czz56\" (UniqueName: \"kubernetes.io/projected/210471f4-8d5c-41ae-afa1-6deccca7846e-kube-api-access-czz56\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-etc-openvswitch\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240673 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-system-cni-dir\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a837857-cbb6-411d-8d9c-1af494b92d13-cni-binary-copy\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97d5427-3253-45fd-8a75-b0f4a8dd0452-tmp-dir\") pod \"node-resolver-qcvmz\" (UID: \"e97d5427-3253-45fd-8a75-b0f4a8dd0452\") " pod="openshift-dns/node-resolver-qcvmz" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-var-lib-kubelet\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240746 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/faf01480-6d08-4434-822f-28b4815084d2-agent-certs\") pod \"konnectivity-agent-9c7xb\" (UID: \"faf01480-6d08-4434-822f-28b4815084d2\") " pod="kube-system/konnectivity-agent-9c7xb" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240763 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/557fb632-fd3a-4b82-b25a-ba2e25072337-cnibin\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/557fb632-fd3a-4b82-b25a-ba2e25072337-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240792 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4532703-7257-43ee-b5bd-fc18dd83a77e-ovnkube-config\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240817 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-run-systemd\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-run-openvswitch\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-sys-fs\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmnfh\" (UniqueName: \"kubernetes.io/projected/e97d5427-3253-45fd-8a75-b0f4a8dd0452-kube-api-access-hmnfh\") pod \"node-resolver-qcvmz\" (UID: \"e97d5427-3253-45fd-8a75-b0f4a8dd0452\") " pod="openshift-dns/node-resolver-qcvmz" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/faf01480-6d08-4434-822f-28b4815084d2-konnectivity-ca\") pod \"konnectivity-agent-9c7xb\" (UID: \"faf01480-6d08-4434-822f-28b4815084d2\") " pod="kube-system/konnectivity-agent-9c7xb" Sep 06 05:53:57.241480 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/557fb632-fd3a-4b82-b25a-ba2e25072337-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240886 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-socket-dir\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-slash\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-run-ovn\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4532703-7257-43ee-b5bd-fc18dd83a77e-env-overrides\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240967 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-var-lib-openvswitch\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-var-lib-cni-multus\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-kubelet\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241013 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-multus-conf-dir\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241032 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-node-log\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241048 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4532703-7257-43ee-b5bd-fc18dd83a77e-ovn-node-metrics-cert\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-multus-cni-dir\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241090 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-run-netns\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241126 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-etc-selinux\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241115 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-hostroot\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmplp\" (UniqueName: \"kubernetes.io/projected/9a837857-cbb6-411d-8d9c-1af494b92d13-kube-api-access-dmplp\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/557fb632-fd3a-4b82-b25a-ba2e25072337-system-cni-dir\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.242279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241200 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/557fb632-fd3a-4b82-b25a-ba2e25072337-cni-binary-copy\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241224 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-run-ovn-kubernetes\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-device-dir\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241254 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-run-k8s-cni-cncf-io\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241329 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-run-k8s-cni-cncf-io\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241359 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-log-socket\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241366 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4532703-7257-43ee-b5bd-fc18dd83a77e-ovnkube-script-lib\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241384 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241411 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-run-multus-certs\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241442 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-systemd-units\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241441 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-run-netns\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241605 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-var-lib-cni-bin\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241616 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-registration-dir\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241651 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-etc-kubernetes\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241693 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-run-ovn\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241696 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241705 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-multus-socket-dir-parent\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.243136 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.240785 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/557fb632-fd3a-4b82-b25a-ba2e25072337-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241761 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e97d5427-3253-45fd-8a75-b0f4a8dd0452-hosts-file\") pod \"node-resolver-qcvmz\" (UID: \"e97d5427-3253-45fd-8a75-b0f4a8dd0452\") " pod="openshift-dns/node-resolver-qcvmz" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241926 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-system-cni-dir\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.241962 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-slash\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242009 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4532703-7257-43ee-b5bd-fc18dd83a77e-env-overrides\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242049 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-run-netns\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242070 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-etc-openvswitch\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242115 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/557fb632-fd3a-4b82-b25a-ba2e25072337-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242151 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/557fb632-fd3a-4b82-b25a-ba2e25072337-system-cni-dir\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242178 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-multus-cni-dir\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242182 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-node-log\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242212 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-device-dir\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242232 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-kubelet\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242246 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-host-run-ovn-kubernetes\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242279 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-var-lib-cni-multus\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242285 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-host-var-lib-kubelet\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242331 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/210471f4-8d5c-41ae-afa1-6deccca7846e-sys-fs\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242360 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-run-systemd\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.243660 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242526 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a837857-cbb6-411d-8d9c-1af494b92d13-cni-binary-copy\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.244370 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242572 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a837857-cbb6-411d-8d9c-1af494b92d13-hostroot\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.244370 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242610 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4532703-7257-43ee-b5bd-fc18dd83a77e-run-openvswitch\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.244370 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242622 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97d5427-3253-45fd-8a75-b0f4a8dd0452-tmp-dir\") pod \"node-resolver-qcvmz\" (UID: \"e97d5427-3253-45fd-8a75-b0f4a8dd0452\") " pod="openshift-dns/node-resolver-qcvmz" Sep 06 05:53:57.244370 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242626 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/557fb632-fd3a-4b82-b25a-ba2e25072337-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.244370 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242678 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/557fb632-fd3a-4b82-b25a-ba2e25072337-cnibin\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.244370 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242681 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/557fb632-fd3a-4b82-b25a-ba2e25072337-cni-binary-copy\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.244370 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242728 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/faf01480-6d08-4434-822f-28b4815084d2-konnectivity-ca\") pod \"konnectivity-agent-9c7xb\" (UID: \"faf01480-6d08-4434-822f-28b4815084d2\") " pod="kube-system/konnectivity-agent-9c7xb" Sep 06 05:53:57.244370 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.242984 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a837857-cbb6-411d-8d9c-1af494b92d13-multus-daemon-config\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.245512 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.245492 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4532703-7257-43ee-b5bd-fc18dd83a77e-ovn-node-metrics-cert\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.245674 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.245558 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/faf01480-6d08-4434-822f-28b4815084d2-agent-certs\") pod \"konnectivity-agent-9c7xb\" (UID: \"faf01480-6d08-4434-822f-28b4815084d2\") " pod="kube-system/konnectivity-agent-9c7xb" Sep 06 05:53:57.247750 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.247709 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 05:53:57.247750 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.247750 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 05:53:57.248024 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.247764 2578 projected.go:196] Error preparing data for projected volume kube-api-access-7bn8x for pod openshift-network-diagnostics/network-check-target-f87xg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:53:57.248024 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.247882 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x podName:a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40 nodeName:}" failed. No retries permitted until 2026-09-06 05:53:57.747862669 +0000 UTC m=+3.146193117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7bn8x" (UniqueName: "kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x") pod "network-check-target-f87xg" (UID: "a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:53:57.249491 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.249416 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czz56\" (UniqueName: \"kubernetes.io/projected/210471f4-8d5c-41ae-afa1-6deccca7846e-kube-api-access-czz56\") pod \"aws-ebs-csi-driver-node-s56fs\" (UID: \"210471f4-8d5c-41ae-afa1-6deccca7846e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.250559 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.250477 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6cn8\" (UniqueName: \"kubernetes.io/projected/557fb632-fd3a-4b82-b25a-ba2e25072337-kube-api-access-v6cn8\") pod \"multus-additional-cni-plugins-4b45g\" (UID: \"557fb632-fd3a-4b82-b25a-ba2e25072337\") " pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.250831 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.250775 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmplp\" (UniqueName: \"kubernetes.io/projected/9a837857-cbb6-411d-8d9c-1af494b92d13-kube-api-access-dmplp\") pod \"multus-dm4ww\" (UID: \"9a837857-cbb6-411d-8d9c-1af494b92d13\") " pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.251108 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.251062 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nm6l\" (UniqueName: \"kubernetes.io/projected/e4532703-7257-43ee-b5bd-fc18dd83a77e-kube-api-access-8nm6l\") pod \"ovnkube-node-dwh56\" (UID: \"e4532703-7257-43ee-b5bd-fc18dd83a77e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.252579 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.252557 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmnfh\" (UniqueName: \"kubernetes.io/projected/e97d5427-3253-45fd-8a75-b0f4a8dd0452-kube-api-access-hmnfh\") pod \"node-resolver-qcvmz\" (UID: \"e97d5427-3253-45fd-8a75-b0f4a8dd0452\") " pod="openshift-dns/node-resolver-qcvmz" Sep 06 05:53:57.325997 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.325963 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dknd9" Sep 06 05:53:57.337338 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:57.337304 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62afc7f8_8ed3_4dbe_95db_e585f70422f8.slice/crio-f4ccdae1e83a112e7547401834e32073a1c36443817637a6a4951cc4891661d1 WatchSource:0}: Error finding container f4ccdae1e83a112e7547401834e32073a1c36443817637a6a4951cc4891661d1: Status 404 returned error can't find the container with id f4ccdae1e83a112e7547401834e32073a1c36443817637a6a4951cc4891661d1 Sep 06 05:53:57.338811 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.338787 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6fw4z" Sep 06 05:53:57.353238 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.353052 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g625l" Sep 06 05:53:57.358929 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.358890 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9c7xb" Sep 06 05:53:57.374678 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.374648 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4b45g" Sep 06 05:53:57.378513 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.378495 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 06 05:53:57.383183 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.383155 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dm4ww" Sep 06 05:53:57.389302 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.389279 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:53:57.397179 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.397157 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" Sep 06 05:53:57.404932 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.404890 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qcvmz" Sep 06 05:53:57.643464 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.643374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:53:57.643464 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.643441 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:53:57.643690 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.643553 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 05:53:57.643690 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.643557 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:53:57.643690 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.643614 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret podName:115882c1-6556-4feb-adc9-a5ddf5f36ba2 nodeName:}" failed. No retries permitted until 2026-09-06 05:53:58.643600371 +0000 UTC m=+4.041930817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret") pod "global-pull-secret-syncer-c66ws" (UID: "115882c1-6556-4feb-adc9-a5ddf5f36ba2") : object "kube-system"/"original-pull-secret" not registered Sep 06 05:53:57.643690 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.643630 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs podName:f36acd55-4ce4-4bdb-9518-cca261651a19 nodeName:}" failed. No retries permitted until 2026-09-06 05:53:58.643623335 +0000 UTC m=+4.041953781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs") pod "network-metrics-daemon-w74pq" (UID: "f36acd55-4ce4-4bdb-9518-cca261651a19") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:53:57.844466 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:57.844413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bn8x\" (UniqueName: \"kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x\") pod \"network-check-target-f87xg\" (UID: \"a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40\") " pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:53:57.844628 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.844596 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 05:53:57.844628 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.844619 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 05:53:57.844724 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.844633 2578 projected.go:196] Error preparing data for projected volume kube-api-access-7bn8x for pod openshift-network-diagnostics/network-check-target-f87xg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:53:57.844724 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:57.844705 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x podName:a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40 nodeName:}" failed. No retries permitted until 2026-09-06 05:53:58.844682735 +0000 UTC m=+4.243013198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7bn8x" (UniqueName: "kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x") pod "network-check-target-f87xg" (UID: "a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:53:58.063629 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:58.063535 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 05:48:57 +0000 UTC" deadline="2028-02-15 02:59:36.340500498 +0000 UTC" Sep 06 05:53:58.063629 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:58.063574 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12645h5m38.27693145s" Sep 06 05:53:58.164454 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:58.164408 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dknd9" event={"ID":"62afc7f8-8ed3-4dbe-95db-e585f70422f8","Type":"ContainerStarted","Data":"f4ccdae1e83a112e7547401834e32073a1c36443817637a6a4951cc4891661d1"} Sep 06 05:53:58.382172 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:58.382127 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbddf45b0_9199_4a9d_aee3_ca253d8a9c56.slice/crio-98ab84ccd8505f37d476ceaa46b85e9d2db6f167516ebe9fdc560407cb2f6f80 WatchSource:0}: Error finding container 98ab84ccd8505f37d476ceaa46b85e9d2db6f167516ebe9fdc560407cb2f6f80: Status 404 returned error can't find the container with id 98ab84ccd8505f37d476ceaa46b85e9d2db6f167516ebe9fdc560407cb2f6f80 Sep 06 05:53:58.401242 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:58.401224 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a837857_cbb6_411d_8d9c_1af494b92d13.slice/crio-be8f7edcc5cacd00ef234c6fdde6d2c804f62e0904e2eee5465b00bbde3821e8 WatchSource:0}: Error finding container be8f7edcc5cacd00ef234c6fdde6d2c804f62e0904e2eee5465b00bbde3821e8: Status 404 returned error can't find the container with id be8f7edcc5cacd00ef234c6fdde6d2c804f62e0904e2eee5465b00bbde3821e8 Sep 06 05:53:58.402843 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:58.402828 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0af07e4_9ff9_462f_a6cc_0d4e9a37de97.slice/crio-2c462dacf8cb6f687a0b21104c31dd8a6de04f513f8ea077bf731dd1f20a8502 WatchSource:0}: Error finding container 2c462dacf8cb6f687a0b21104c31dd8a6de04f513f8ea077bf731dd1f20a8502: Status 404 returned error can't find the container with id 2c462dacf8cb6f687a0b21104c31dd8a6de04f513f8ea077bf731dd1f20a8502 Sep 06 05:53:58.404661 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:53:58.404421 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210471f4_8d5c_41ae_afa1_6deccca7846e.slice/crio-eea2988ee3031cf0505f240652c7b64e934364cdcb93b68304a7729c83ae6a90 WatchSource:0}: Error finding container eea2988ee3031cf0505f240652c7b64e934364cdcb93b68304a7729c83ae6a90: Status 404 returned error can't find the container with id eea2988ee3031cf0505f240652c7b64e934364cdcb93b68304a7729c83ae6a90 Sep 06 05:53:58.651878 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:58.651753 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:53:58.651878 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:58.651811 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:53:58.652073 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:58.651945 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:53:58.652073 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:58.651959 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 05:53:58.652073 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:58.652030 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret podName:115882c1-6556-4feb-adc9-a5ddf5f36ba2 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:00.652010046 +0000 UTC m=+6.050340498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret") pod "global-pull-secret-syncer-c66ws" (UID: "115882c1-6556-4feb-adc9-a5ddf5f36ba2") : object "kube-system"/"original-pull-secret" not registered Sep 06 05:53:58.652073 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:58.652050 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs podName:f36acd55-4ce4-4bdb-9518-cca261651a19 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:00.65203937 +0000 UTC m=+6.050369823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs") pod "network-metrics-daemon-w74pq" (UID: "f36acd55-4ce4-4bdb-9518-cca261651a19") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:53:58.854212 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:58.853880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bn8x\" (UniqueName: \"kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x\") pod \"network-check-target-f87xg\" (UID: \"a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40\") " pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:53:58.854392 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:58.854380 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 05:53:58.854452 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:58.854401 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 05:53:58.854452 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:58.854414 2578 projected.go:196] Error preparing data for projected volume kube-api-access-7bn8x for pod openshift-network-diagnostics/network-check-target-f87xg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:53:58.854568 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:58.854475 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x podName:a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:00.854455458 +0000 UTC m=+6.252785909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7bn8x" (UniqueName: "kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x") pod "network-check-target-f87xg" (UID: "a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:53:59.064099 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.063967 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 05:48:57 +0000 UTC" deadline="2028-04-10 04:43:11.544697641 +0000 UTC" Sep 06 05:53:59.064099 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.064031 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13966h49m12.480670781s" Sep 06 05:53:59.142292 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.141456 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:53:59.142292 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:59.141585 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:53:59.142292 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.142010 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:53:59.142292 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:59.142115 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:53:59.142292 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.142173 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:53:59.142292 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:53:59.142239 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:53:59.174179 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.174139 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9c7xb" event={"ID":"faf01480-6d08-4434-822f-28b4815084d2","Type":"ContainerStarted","Data":"fafedd380dd0b884784bacaa1705599bb279df478991b3aba70a2ce6a5c02127"} Sep 06 05:53:59.183039 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.182990 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" event={"ID":"e4532703-7257-43ee-b5bd-fc18dd83a77e","Type":"ContainerStarted","Data":"608b7824c7d73d6e0ecfd7794a49df1acef8c7feb0cd60ea7fb3cb2b4b5b9074"} Sep 06 05:53:59.197607 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.197571 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4b45g" event={"ID":"557fb632-fd3a-4b82-b25a-ba2e25072337","Type":"ContainerStarted","Data":"c14e63d8ccf14eb4788a220b8ed043c523187b3276e09f966916a5778ea0319d"} Sep 06 05:53:59.218498 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.218455 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" event={"ID":"210471f4-8d5c-41ae-afa1-6deccca7846e","Type":"ContainerStarted","Data":"eea2988ee3031cf0505f240652c7b64e934364cdcb93b68304a7729c83ae6a90"} Sep 06 05:53:59.236102 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.236062 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g625l" event={"ID":"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97","Type":"ContainerStarted","Data":"2c462dacf8cb6f687a0b21104c31dd8a6de04f513f8ea077bf731dd1f20a8502"} Sep 06 05:53:59.252929 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.252221 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" event={"ID":"bd1de5c49e7422a298303b9a2c5c9a84","Type":"ContainerStarted","Data":"04a9071a4b776ae17c35ba121d31309e8ae6b46e559c9c5aff0b28b151251afe"} Sep 06 05:53:59.270234 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.270141 2578 generic.go:356] "Generic (PLEG): container finished" podID="4b74d93d1db0c298014dfe267b0549d7" containerID="b076cf6a0ff4e3c70ae66dc8c36d7a2d7c2078e369c2510d6e68b15a8fc12224" exitCode=0 Sep 06 05:53:59.270420 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.270282 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" event={"ID":"4b74d93d1db0c298014dfe267b0549d7","Type":"ContainerDied","Data":"b076cf6a0ff4e3c70ae66dc8c36d7a2d7c2078e369c2510d6e68b15a8fc12224"} Sep 06 05:53:59.289377 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.289315 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-32.ec2.internal" podStartSLOduration=3.289290371 podStartE2EDuration="3.289290371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:53:59.267037189 +0000 UTC m=+4.665367676" watchObservedRunningTime="2026-09-06 05:53:59.289290371 +0000 UTC m=+4.687620840" Sep 06 05:53:59.299107 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.299072 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qcvmz" event={"ID":"e97d5427-3253-45fd-8a75-b0f4a8dd0452","Type":"ContainerStarted","Data":"df6ec803687a7cb33dd5759f042ab9324aa6a3e6df95d392d74a8499b3a0c611"} Sep 06 05:53:59.312630 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.312572 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dm4ww" event={"ID":"9a837857-cbb6-411d-8d9c-1af494b92d13","Type":"ContainerStarted","Data":"be8f7edcc5cacd00ef234c6fdde6d2c804f62e0904e2eee5465b00bbde3821e8"} Sep 06 05:53:59.326895 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:53:59.326769 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6fw4z" event={"ID":"bddf45b0-9199-4a9d-aee3-ca253d8a9c56","Type":"ContainerStarted","Data":"98ab84ccd8505f37d476ceaa46b85e9d2db6f167516ebe9fdc560407cb2f6f80"} Sep 06 05:54:00.338469 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:00.338310 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" event={"ID":"4b74d93d1db0c298014dfe267b0549d7","Type":"ContainerStarted","Data":"8856aa6cebadb8d2552e31a00a72816c90f58f26db0fc981f0606287cf9691fd"} Sep 06 05:54:00.670657 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:00.670577 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:00.670657 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:00.670657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:00.670889 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:00.670785 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:00.670889 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:00.670843 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs podName:f36acd55-4ce4-4bdb-9518-cca261651a19 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:04.670828697 +0000 UTC m=+10.069159143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs") pod "network-metrics-daemon-w74pq" (UID: "f36acd55-4ce4-4bdb-9518-cca261651a19") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:00.671253 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:00.671236 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:00.671319 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:00.671294 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret podName:115882c1-6556-4feb-adc9-a5ddf5f36ba2 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:04.671278827 +0000 UTC m=+10.069609278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret") pod "global-pull-secret-syncer-c66ws" (UID: "115882c1-6556-4feb-adc9-a5ddf5f36ba2") : object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:00.872771 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:00.872732 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bn8x\" (UniqueName: \"kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x\") pod \"network-check-target-f87xg\" (UID: \"a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40\") " pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:00.872980 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:00.872897 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 05:54:00.872980 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:00.872935 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 05:54:00.872980 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:00.872948 2578 projected.go:196] Error preparing data for projected volume kube-api-access-7bn8x for pod openshift-network-diagnostics/network-check-target-f87xg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:00.873139 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:00.873005 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x podName:a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:04.872986066 +0000 UTC m=+10.271316513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7bn8x" (UniqueName: "kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x") pod "network-check-target-f87xg" (UID: "a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:01.141318 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:01.141235 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:01.141485 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:01.141364 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:01.141744 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:01.141729 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:01.141840 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:01.141806 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:01.141919 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:01.141851 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:01.141967 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:01.141916 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:03.139029 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:03.138994 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:03.139472 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:03.139126 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:03.139472 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:03.139141 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:03.139472 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:03.139244 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:03.139472 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:03.138994 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:03.139472 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:03.139341 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:04.708564 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:04.708520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:04.709013 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:04.708620 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:04.709013 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:04.708683 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:04.709013 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:04.708745 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:04.709013 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:04.708758 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret podName:115882c1-6556-4feb-adc9-a5ddf5f36ba2 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:12.708738417 +0000 UTC m=+18.107068876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret") pod "global-pull-secret-syncer-c66ws" (UID: "115882c1-6556-4feb-adc9-a5ddf5f36ba2") : object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:04.709013 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:04.708797 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs podName:f36acd55-4ce4-4bdb-9518-cca261651a19 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:12.708783596 +0000 UTC m=+18.107114042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs") pod "network-metrics-daemon-w74pq" (UID: "f36acd55-4ce4-4bdb-9518-cca261651a19") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:04.910423 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:04.910373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bn8x\" (UniqueName: \"kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x\") pod \"network-check-target-f87xg\" (UID: \"a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40\") " pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:04.910599 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:04.910567 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 05:54:04.910671 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:04.910601 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 05:54:04.910671 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:04.910618 2578 projected.go:196] Error preparing data for projected volume kube-api-access-7bn8x for pod openshift-network-diagnostics/network-check-target-f87xg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:04.910734 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:04.910680 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x podName:a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:12.910661565 +0000 UTC m=+18.308992030 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7bn8x" (UniqueName: "kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x") pod "network-check-target-f87xg" (UID: "a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:05.139628 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:05.139547 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:05.139801 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:05.139658 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:05.140125 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:05.139984 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:05.140125 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:05.140076 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:05.140125 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:05.140073 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:05.140326 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:05.140166 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:07.139067 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:07.138981 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:07.139067 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:07.139007 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:07.139067 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:07.139007 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:07.139624 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:07.139096 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:07.139624 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:07.139197 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:07.139624 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:07.139284 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:09.138854 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:09.138816 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:09.139283 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:09.138813 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:09.139283 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:09.138995 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:09.139283 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:09.138813 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:09.139283 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:09.139038 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:09.139283 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:09.139136 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:11.138262 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:11.138222 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:11.138262 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:11.138260 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:11.138755 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:11.138222 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:11.138755 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:11.138361 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:11.138755 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:11.138428 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:11.138755 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:11.138511 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:12.767258 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:12.767220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:12.767729 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:12.767283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:12.767729 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:12.767398 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:12.767729 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:12.767447 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:12.767729 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:12.767483 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs podName:f36acd55-4ce4-4bdb-9518-cca261651a19 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:28.767461356 +0000 UTC m=+34.165791805 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs") pod "network-metrics-daemon-w74pq" (UID: "f36acd55-4ce4-4bdb-9518-cca261651a19") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:12.767729 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:12.767511 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret podName:115882c1-6556-4feb-adc9-a5ddf5f36ba2 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:28.7674918 +0000 UTC m=+34.165822258 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret") pod "global-pull-secret-syncer-c66ws" (UID: "115882c1-6556-4feb-adc9-a5ddf5f36ba2") : object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:12.968975 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:12.968934 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bn8x\" (UniqueName: \"kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x\") pod \"network-check-target-f87xg\" (UID: \"a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40\") " pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:12.969154 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:12.969132 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 05:54:12.969200 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:12.969166 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 05:54:12.969200 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:12.969181 2578 projected.go:196] Error preparing data for projected volume kube-api-access-7bn8x for pod openshift-network-diagnostics/network-check-target-f87xg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:12.969283 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:12.969253 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x podName:a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40 nodeName:}" failed. No retries permitted until 2026-09-06 05:54:28.969232827 +0000 UTC m=+34.367563288 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7bn8x" (UniqueName: "kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x") pod "network-check-target-f87xg" (UID: "a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:13.138823 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:13.138747 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:13.138823 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:13.138765 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:13.138823 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:13.138765 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:13.139100 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:13.138877 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:13.139100 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:13.138964 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:13.139100 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:13.139045 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:15.139608 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:15.139569 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:15.140036 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:15.139683 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:15.140036 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:15.139712 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:15.140036 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:15.139774 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:15.140036 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:15.139789 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:15.140036 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:15.139893 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:17.140687 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.138993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:17.140687 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.138993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:17.140687 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:17.139332 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:17.140687 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.138993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:17.140687 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:17.139384 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:17.140687 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:17.139478 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:17.380187 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.379921 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" event={"ID":"210471f4-8d5c-41ae-afa1-6deccca7846e","Type":"ContainerStarted","Data":"9d8533ff608ce3b2c1c55b71afe61d462c0d3150fb0b06882bfa1908b94da95d"} Sep 06 05:54:17.381659 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.381626 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g625l" event={"ID":"f0af07e4-9ff9-462f-a6cc-0d4e9a37de97","Type":"ContainerStarted","Data":"60153cd8b39bc417a7eff2dca7bf3629b22696e5ef7c7bef52261a7cbee9e330"} Sep 06 05:54:17.383263 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.383233 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qcvmz" event={"ID":"e97d5427-3253-45fd-8a75-b0f4a8dd0452","Type":"ContainerStarted","Data":"5d67a43176d57f027ff17fa7bda58d08e9d435bea66b65509926d20244afc8b1"} Sep 06 05:54:17.384852 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.384812 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dm4ww" event={"ID":"9a837857-cbb6-411d-8d9c-1af494b92d13","Type":"ContainerStarted","Data":"b56fc09f32733097d756875b67b2c092b36ea18c2c29b556e549477542289981"} Sep 06 05:54:17.386599 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.386579 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6fw4z" event={"ID":"bddf45b0-9199-4a9d-aee3-ca253d8a9c56","Type":"ContainerStarted","Data":"5ae0287601b801febd39ca73a4008d04d075811fe77433cf18771bf93a72aa5c"} Sep 06 05:54:17.388237 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.388217 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9c7xb" event={"ID":"faf01480-6d08-4434-822f-28b4815084d2","Type":"ContainerStarted","Data":"46f34f8a9a42d7e1fc46cdd471a524a207c6171b15cc808b25d7367cba8ec954"} Sep 06 05:54:17.391546 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.391496 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" event={"ID":"e4532703-7257-43ee-b5bd-fc18dd83a77e","Type":"ContainerStarted","Data":"24c2935c44ac59f3bf85dfbcf02b3c2d087eaed878e751e72d9f529cc9e3c6cc"} Sep 06 05:54:17.391638 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.391549 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" event={"ID":"e4532703-7257-43ee-b5bd-fc18dd83a77e","Type":"ContainerStarted","Data":"223278f3a6b1411f92929a6182d6328d345a8f541d1ecd355504e981f944cc5a"} Sep 06 05:54:17.391638 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.391595 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" event={"ID":"e4532703-7257-43ee-b5bd-fc18dd83a77e","Type":"ContainerStarted","Data":"5c53bee8712ee94d0bba385fc7c7824b846f5593c064d8012d7d2beb2fc2aa18"} Sep 06 05:54:17.391638 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.391637 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" event={"ID":"e4532703-7257-43ee-b5bd-fc18dd83a77e","Type":"ContainerStarted","Data":"de4f69d68f123d8145759e39fd246ab46f325b79feebfd6be24a26290215b0a5"} Sep 06 05:54:17.391789 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.391680 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" event={"ID":"e4532703-7257-43ee-b5bd-fc18dd83a77e","Type":"ContainerStarted","Data":"926da5ef908138c2178e7b100ab5081b5f88759944de79505e729324db70d5ff"} Sep 06 05:54:17.391789 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.391728 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" event={"ID":"e4532703-7257-43ee-b5bd-fc18dd83a77e","Type":"ContainerStarted","Data":"dafcecff54fd2ad3a6ab592756bd48887f1d1cc9d77dead47e7c5bf8a83cfab4"} Sep 06 05:54:17.393228 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.393204 2578 generic.go:356] "Generic (PLEG): container finished" podID="557fb632-fd3a-4b82-b25a-ba2e25072337" containerID="d165972ca3656ec00a69d0bde9ee59bfcd08439618ebb051a08217a63ce3a4e7" exitCode=0 Sep 06 05:54:17.393318 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.393238 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4b45g" event={"ID":"557fb632-fd3a-4b82-b25a-ba2e25072337","Type":"ContainerDied","Data":"d165972ca3656ec00a69d0bde9ee59bfcd08439618ebb051a08217a63ce3a4e7"} Sep 06 05:54:17.396546 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.396502 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-32.ec2.internal" podStartSLOduration=21.396489784 podStartE2EDuration="21.396489784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:00.353686991 +0000 UTC m=+5.752017485" watchObservedRunningTime="2026-09-06 05:54:17.396489784 +0000 UTC m=+22.794820252" Sep 06 05:54:17.396786 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.396758 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g625l" podStartSLOduration=4.554562867 podStartE2EDuration="22.396751146s" totalImagesPullingTime="17.842188279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:17.395952468 +0000 UTC m=+22.794282947" watchObservedRunningTime="2026-09-06 05:54:17.396751146 +0000 UTC m=+22.795081633" Sep 06 05:54:17.411740 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.411697 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9c7xb" podStartSLOduration=4.580227604 podStartE2EDuration="22.411678271s" totalImagesPullingTime="17.831450667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:17.411519825 +0000 UTC m=+22.809850294" watchObservedRunningTime="2026-09-06 05:54:17.411678271 +0000 UTC m=+22.810008740" Sep 06 05:54:17.442504 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.442454 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qcvmz" podStartSLOduration=8.963386568 podStartE2EDuration="22.442436984s" totalImagesPullingTime="13.479050416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:17.441873412 +0000 UTC m=+22.840203901" watchObservedRunningTime="2026-09-06 05:54:17.442436984 +0000 UTC m=+22.840767452" Sep 06 05:54:17.469091 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.469038 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6fw4z" podStartSLOduration=4.611546442 podStartE2EDuration="22.469020423s" totalImagesPullingTime="17.857473981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:17.454093338 +0000 UTC m=+22.852423806" watchObservedRunningTime="2026-09-06 05:54:17.469020423 +0000 UTC m=+22.867350891" Sep 06 05:54:17.469281 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.469255 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dm4ww" podStartSLOduration=4.384621201 podStartE2EDuration="22.469243689s" totalImagesPullingTime="18.084622488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:17.468944084 +0000 UTC m=+22.867274576" watchObservedRunningTime="2026-09-06 05:54:17.469243689 +0000 UTC m=+22.867574157" Sep 06 05:54:17.797743 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:17.797718 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 06 05:54:18.106779 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:18.106521 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-06T05:54:17.797737529Z","UUID":"a38b3dd5-4ca7-4d83-854b-7f0af3a75ca5","Handler":null,"Name":"","Endpoint":""} Sep 06 05:54:18.108541 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:18.108516 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 06 05:54:18.108652 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:18.108553 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 06 05:54:18.397745 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:18.397659 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" event={"ID":"210471f4-8d5c-41ae-afa1-6deccca7846e","Type":"ContainerStarted","Data":"530f3705e81f5eba05fc8c30ae53fa16c0576c35b1e33f0ac730d4d7230ff344"} Sep 06 05:54:18.399290 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:18.399258 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dknd9" event={"ID":"62afc7f8-8ed3-4dbe-95db-e585f70422f8","Type":"ContainerStarted","Data":"ab0a66f21edd84187027c07ddb9cb438f8e8f3b4bd806c2c4681219e8c86df89"} Sep 06 05:54:18.411917 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:18.411862 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dknd9" podStartSLOduration=8.8638032 podStartE2EDuration="23.411846572s" totalImagesPullingTime="14.548043372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:18.411559989 +0000 UTC m=+23.809890482" watchObservedRunningTime="2026-09-06 05:54:18.411846572 +0000 UTC m=+23.810177039" Sep 06 05:54:19.138927 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:19.138875 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:19.139108 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:19.138880 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:19.139108 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:19.139025 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:19.139108 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:19.139066 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:19.139271 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:19.139175 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:19.139322 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:19.139267 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:19.161095 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:19.161046 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9c7xb" Sep 06 05:54:19.161648 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:19.161630 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9c7xb" Sep 06 05:54:19.406017 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:19.405923 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" event={"ID":"e4532703-7257-43ee-b5bd-fc18dd83a77e","Type":"ContainerStarted","Data":"c235c106c66467e4dd03d4f059f05b889ffc18ecd08ca24e6386970acff671cb"} Sep 06 05:54:19.408192 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:19.408162 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" event={"ID":"210471f4-8d5c-41ae-afa1-6deccca7846e","Type":"ContainerStarted","Data":"3d4922ef7d65d1a9ccf8ab862ff508490227291995aa9e99fe4ae5e9b35516a2"} Sep 06 05:54:19.408698 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:19.408674 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9c7xb" Sep 06 05:54:19.409148 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:19.409124 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9c7xb" Sep 06 05:54:19.425317 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:19.425258 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s56fs" podStartSLOduration=4.224780065 podStartE2EDuration="24.425242109s" totalImagesPullingTime="20.200462044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:19.4250642 +0000 UTC m=+24.823394685" watchObservedRunningTime="2026-09-06 05:54:19.425242109 +0000 UTC m=+24.823572577" Sep 06 05:54:21.138925 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:21.138870 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:21.139380 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:21.138879 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:21.139380 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:21.139021 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:21.139380 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:21.138878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:21.139380 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:21.139085 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:21.139380 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:21.139161 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:22.418887 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:22.418647 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" event={"ID":"e4532703-7257-43ee-b5bd-fc18dd83a77e","Type":"ContainerStarted","Data":"0d85081a9ca889c9430e14985a5956fa36420e9e47a79c9c8fd69e074d3b0ade"} Sep 06 05:54:22.419512 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:22.418975 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:54:22.421312 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:22.421285 2578 generic.go:356] "Generic (PLEG): container finished" podID="557fb632-fd3a-4b82-b25a-ba2e25072337" containerID="fcf662bd406dffba7c353ba9679705142a7e6ab15f8f482a1ac84c5473751593" exitCode=0 Sep 06 05:54:22.421433 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:22.421323 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4b45g" event={"ID":"557fb632-fd3a-4b82-b25a-ba2e25072337","Type":"ContainerDied","Data":"fcf662bd406dffba7c353ba9679705142a7e6ab15f8f482a1ac84c5473751593"} Sep 06 05:54:22.436073 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:22.436050 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:54:22.444552 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:22.444509 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" podStartSLOduration=9.606177105 podStartE2EDuration="27.444496654s" totalImagesPullingTime="17.838319549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:22.444066359 +0000 UTC m=+27.842396848" watchObservedRunningTime="2026-09-06 05:54:22.444496654 +0000 UTC m=+27.842827144" Sep 06 05:54:23.138700 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:23.138668 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:23.138966 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:23.138668 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:23.138966 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:23.138778 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:23.138966 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:23.138792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:23.138966 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:23.138891 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:23.139247 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:23.139025 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:23.424799 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:23.424551 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:54:23.424799 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:23.424774 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:54:23.440111 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:23.440082 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:54:23.548267 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:23.548231 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c66ws"] Sep 06 05:54:23.548436 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:23.548357 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:23.548496 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:23.548474 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:23.551215 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:23.551187 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w74pq"] Sep 06 05:54:23.551403 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:23.551313 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:23.551465 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:23.551421 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:23.551839 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:23.551816 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f87xg"] Sep 06 05:54:23.551959 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:23.551924 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:23.552044 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:23.552023 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:24.428799 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:24.428767 2578 generic.go:356] "Generic (PLEG): container finished" podID="557fb632-fd3a-4b82-b25a-ba2e25072337" containerID="19d8712660facbd59056634af64f38fe72a2967c985695d0bccc73d8f38a5ef9" exitCode=0 Sep 06 05:54:24.429212 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:24.428860 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4b45g" event={"ID":"557fb632-fd3a-4b82-b25a-ba2e25072337","Type":"ContainerDied","Data":"19d8712660facbd59056634af64f38fe72a2967c985695d0bccc73d8f38a5ef9"} Sep 06 05:54:25.139464 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:25.139405 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:25.139626 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:25.139511 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:25.139626 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:25.139530 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:25.139626 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:25.139585 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:25.139755 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:25.139677 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:25.139804 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:25.139773 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:26.436139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:26.436104 2578 generic.go:356] "Generic (PLEG): container finished" podID="557fb632-fd3a-4b82-b25a-ba2e25072337" containerID="e03ecc820a6173efd93b1adea84c93e3d5c2a2250baecba06aa5c1aeaade6f90" exitCode=0 Sep 06 05:54:26.436529 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:26.436161 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4b45g" event={"ID":"557fb632-fd3a-4b82-b25a-ba2e25072337","Type":"ContainerDied","Data":"e03ecc820a6173efd93b1adea84c93e3d5c2a2250baecba06aa5c1aeaade6f90"} Sep 06 05:54:27.138699 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:27.138658 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:27.138699 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:27.138695 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:27.138933 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:27.138663 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:27.138933 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:27.138761 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:27.138933 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:27.138865 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:27.139048 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:27.138959 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:28.351329 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:28.351101 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c7xb_faf01480-6d08-4434-822f-28b4815084d2/konnectivity-agent/0.log" limit=1024 Sep 06 05:54:28.436794 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:28.436765 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9c7xb_faf01480-6d08-4434-822f-28b4815084d2/konnectivity-agent/0.log" limit=1024 Sep 06 05:54:28.790713 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:28.790614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:28.790713 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:28.790701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:28.791089 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:28.790787 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:28.791089 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:28.790787 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:28.791089 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:28.790855 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs podName:f36acd55-4ce4-4bdb-9518-cca261651a19 nodeName:}" failed. No retries permitted until 2026-09-06 05:55:00.790837811 +0000 UTC m=+66.189168262 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs") pod "network-metrics-daemon-w74pq" (UID: "f36acd55-4ce4-4bdb-9518-cca261651a19") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 06 05:54:28.791089 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:28.790936 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret podName:115882c1-6556-4feb-adc9-a5ddf5f36ba2 nodeName:}" failed. No retries permitted until 2026-09-06 05:55:00.790914631 +0000 UTC m=+66.189245091 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret") pod "global-pull-secret-syncer-c66ws" (UID: "115882c1-6556-4feb-adc9-a5ddf5f36ba2") : object "kube-system"/"original-pull-secret" not registered Sep 06 05:54:28.992686 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:28.992640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bn8x\" (UniqueName: \"kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x\") pod \"network-check-target-f87xg\" (UID: \"a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40\") " pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:28.992861 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:28.992826 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 06 05:54:28.992861 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:28.992850 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 06 05:54:28.992999 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:28.992864 2578 projected.go:196] Error preparing data for projected volume kube-api-access-7bn8x for pod openshift-network-diagnostics/network-check-target-f87xg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:28.992999 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:28.992951 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x podName:a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40 nodeName:}" failed. No retries permitted until 2026-09-06 05:55:00.992931485 +0000 UTC m=+66.391261944 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7bn8x" (UniqueName: "kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x") pod "network-check-target-f87xg" (UID: "a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 06 05:54:29.138970 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.138876 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:29.139138 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.138875 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:29.139138 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:29.139037 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:54:29.139138 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.139075 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:29.139303 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:29.139181 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f87xg" podUID="a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40" Sep 06 05:54:29.139359 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:29.139303 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c66ws" podUID="115882c1-6556-4feb-adc9-a5ddf5f36ba2" Sep 06 05:54:29.400265 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.400191 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-32.ec2.internal" event="NodeReady" Sep 06 05:54:29.400677 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.400307 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 06 05:54:29.432941 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.432896 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f46956478-tn984"] Sep 06 05:54:29.437736 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.437709 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.440050 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.440017 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 06 05:54:29.440186 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.440117 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 06 05:54:29.440282 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.440267 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 06 05:54:29.440497 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.440347 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qcp4t\"" Sep 06 05:54:29.443618 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.443593 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zmlqw"] Sep 06 05:54:29.445753 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.445669 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 06 05:54:29.447372 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.447349 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-trktn"] Sep 06 05:54:29.447699 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.447683 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:29.450359 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.450337 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 06 05:54:29.450563 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.450349 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 06 05:54:29.450696 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.450583 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ppscw\"" Sep 06 05:54:29.451797 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.451776 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f46956478-tn984"] Sep 06 05:54:29.451927 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.451874 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-trktn" Sep 06 05:54:29.454230 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.453737 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 06 05:54:29.454230 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.453985 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hqtmv\"" Sep 06 05:54:29.454230 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.454027 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 06 05:54:29.454230 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.453984 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 06 05:54:29.458751 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.458733 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zmlqw"] Sep 06 05:54:29.459137 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.459103 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-trktn"] Sep 06 05:54:29.598106 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.598069 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee5736d-8984-42af-9121-8fe5d123a3ce-tmp-dir\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:29.598299 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.598123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5edfb5f6-d9a0-4e21-8195-6be1703189fb-installation-pull-secrets\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.598299 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.598186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84dj8\" (UniqueName: \"kubernetes.io/projected/fee5736d-8984-42af-9121-8fe5d123a3ce-kube-api-access-84dj8\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:29.598299 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.598272 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fee5736d-8984-42af-9121-8fe5d123a3ce-config-volume\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:29.598465 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.598304 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:29.598465 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.598344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.598465 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.598412 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5edfb5f6-d9a0-4e21-8195-6be1703189fb-ca-trust-extracted\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.598465 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.598435 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-certificates\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.598465 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.598453 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5edfb5f6-d9a0-4e21-8195-6be1703189fb-trusted-ca\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.598657 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.598517 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5edfb5f6-d9a0-4e21-8195-6be1703189fb-image-registry-private-configuration\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.598657 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.598545 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-bound-sa-token\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.598657 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.598584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-487n2\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-kube-api-access-487n2\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.598657 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.598601 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwn8z\" (UniqueName: \"kubernetes.io/projected/df6097ca-4ae8-47d6-918b-189f75f655cd-kube-api-access-nwn8z\") pod \"ingress-canary-trktn\" (UID: \"df6097ca-4ae8-47d6-918b-189f75f655cd\") " pod="openshift-ingress-canary/ingress-canary-trktn" Sep 06 05:54:29.598657 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.598629 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df6097ca-4ae8-47d6-918b-189f75f655cd-cert\") pod \"ingress-canary-trktn\" (UID: \"df6097ca-4ae8-47d6-918b-189f75f655cd\") " pod="openshift-ingress-canary/ingress-canary-trktn" Sep 06 05:54:29.699444 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.699360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.699444 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.699427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5edfb5f6-d9a0-4e21-8195-6be1703189fb-ca-trust-extracted\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.699701 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.699455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-certificates\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.699701 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.699478 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5edfb5f6-d9a0-4e21-8195-6be1703189fb-trusted-ca\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.699701 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.699515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5edfb5f6-d9a0-4e21-8195-6be1703189fb-image-registry-private-configuration\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.699701 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.699532 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-bound-sa-token\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.699701 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:29.699543 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:54:29.699701 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:29.699566 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f46956478-tn984: secret "image-registry-tls" not found Sep 06 05:54:29.699701 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:29.699643 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls podName:5edfb5f6-d9a0-4e21-8195-6be1703189fb nodeName:}" failed. No retries permitted until 2026-09-06 05:54:30.199622583 +0000 UTC m=+35.597953046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls") pod "image-registry-6f46956478-tn984" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb") : secret "image-registry-tls" not found Sep 06 05:54:29.699701 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.699550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-487n2\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-kube-api-access-487n2\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.699701 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.699691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwn8z\" (UniqueName: \"kubernetes.io/projected/df6097ca-4ae8-47d6-918b-189f75f655cd-kube-api-access-nwn8z\") pod \"ingress-canary-trktn\" (UID: \"df6097ca-4ae8-47d6-918b-189f75f655cd\") " pod="openshift-ingress-canary/ingress-canary-trktn" Sep 06 05:54:29.700173 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.699726 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df6097ca-4ae8-47d6-918b-189f75f655cd-cert\") pod \"ingress-canary-trktn\" (UID: \"df6097ca-4ae8-47d6-918b-189f75f655cd\") " pod="openshift-ingress-canary/ingress-canary-trktn" Sep 06 05:54:29.700173 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.699766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee5736d-8984-42af-9121-8fe5d123a3ce-tmp-dir\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:29.700173 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.699793 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5edfb5f6-d9a0-4e21-8195-6be1703189fb-installation-pull-secrets\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.700173 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.699815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84dj8\" (UniqueName: \"kubernetes.io/projected/fee5736d-8984-42af-9121-8fe5d123a3ce-kube-api-access-84dj8\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:29.700173 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.699851 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fee5736d-8984-42af-9121-8fe5d123a3ce-config-volume\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:29.700173 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.699882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:29.700173 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.699967 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5edfb5f6-d9a0-4e21-8195-6be1703189fb-ca-trust-extracted\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.700522 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.700222 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-certificates\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.700522 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.700252 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee5736d-8984-42af-9121-8fe5d123a3ce-tmp-dir\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:29.700522 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:29.700319 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:54:29.700522 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:29.700383 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls podName:fee5736d-8984-42af-9121-8fe5d123a3ce nodeName:}" failed. No retries permitted until 2026-09-06 05:54:30.20035739 +0000 UTC m=+35.598687863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls") pod "dns-default-zmlqw" (UID: "fee5736d-8984-42af-9121-8fe5d123a3ce") : secret "dns-default-metrics-tls" not found Sep 06 05:54:29.700722 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.700570 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5edfb5f6-d9a0-4e21-8195-6be1703189fb-trusted-ca\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.700836 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.700815 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fee5736d-8984-42af-9121-8fe5d123a3ce-config-volume\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:29.704346 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.704316 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5edfb5f6-d9a0-4e21-8195-6be1703189fb-installation-pull-secrets\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.704346 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.704329 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df6097ca-4ae8-47d6-918b-189f75f655cd-cert\") pod \"ingress-canary-trktn\" (UID: \"df6097ca-4ae8-47d6-918b-189f75f655cd\") " pod="openshift-ingress-canary/ingress-canary-trktn" Sep 06 05:54:29.704529 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.704348 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5edfb5f6-d9a0-4e21-8195-6be1703189fb-image-registry-private-configuration\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.710695 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.710667 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84dj8\" (UniqueName: \"kubernetes.io/projected/fee5736d-8984-42af-9121-8fe5d123a3ce-kube-api-access-84dj8\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:29.711392 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.711026 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-bound-sa-token\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.711392 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.711351 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwn8z\" (UniqueName: \"kubernetes.io/projected/df6097ca-4ae8-47d6-918b-189f75f655cd-kube-api-access-nwn8z\") pod \"ingress-canary-trktn\" (UID: \"df6097ca-4ae8-47d6-918b-189f75f655cd\") " pod="openshift-ingress-canary/ingress-canary-trktn" Sep 06 05:54:29.711630 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.711607 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-487n2\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-kube-api-access-487n2\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:29.769764 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:29.769726 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-trktn" Sep 06 05:54:30.203608 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:30.203573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:30.203608 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:30.203612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:30.203834 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:30.203730 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:54:30.203834 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:30.203764 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:54:30.203834 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:30.203781 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f46956478-tn984: secret "image-registry-tls" not found Sep 06 05:54:30.203834 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:30.203813 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls podName:fee5736d-8984-42af-9121-8fe5d123a3ce nodeName:}" failed. No retries permitted until 2026-09-06 05:54:31.203794425 +0000 UTC m=+36.602124898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls") pod "dns-default-zmlqw" (UID: "fee5736d-8984-42af-9121-8fe5d123a3ce") : secret "dns-default-metrics-tls" not found Sep 06 05:54:30.203834 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:30.203829 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls podName:5edfb5f6-d9a0-4e21-8195-6be1703189fb nodeName:}" failed. No retries permitted until 2026-09-06 05:54:31.20382184 +0000 UTC m=+36.602152286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls") pod "image-registry-6f46956478-tn984" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb") : secret "image-registry-tls" not found Sep 06 05:54:30.781948 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:30.781567 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-trktn"] Sep 06 05:54:30.794766 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:54:30.794735 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6097ca_4ae8_47d6_918b_189f75f655cd.slice/crio-9c8d6d9cb9185052061477104dfbe60f231d2354b157785402b549e4503f299d WatchSource:0}: Error finding container 9c8d6d9cb9185052061477104dfbe60f231d2354b157785402b549e4503f299d: Status 404 returned error can't find the container with id 9c8d6d9cb9185052061477104dfbe60f231d2354b157785402b549e4503f299d Sep 06 05:54:31.138801 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:31.138713 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:54:31.138979 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:31.138717 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:54:31.138979 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:31.138717 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:54:31.141324 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:31.141300 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 06 05:54:31.141433 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:31.141307 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n9n5z\"" Sep 06 05:54:31.142249 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:31.142230 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 06 05:54:31.142339 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:31.142230 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 06 05:54:31.142339 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:31.142306 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9rnw9\"" Sep 06 05:54:31.142457 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:31.142355 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 06 05:54:31.211161 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:31.211100 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:31.211161 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:31.211163 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:31.211396 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:31.211271 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:54:31.211396 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:31.211327 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:54:31.211396 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:31.211341 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f46956478-tn984: secret "image-registry-tls" not found Sep 06 05:54:31.211396 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:31.211353 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls podName:fee5736d-8984-42af-9121-8fe5d123a3ce nodeName:}" failed. No retries permitted until 2026-09-06 05:54:33.211332071 +0000 UTC m=+38.609662522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls") pod "dns-default-zmlqw" (UID: "fee5736d-8984-42af-9121-8fe5d123a3ce") : secret "dns-default-metrics-tls" not found Sep 06 05:54:31.211543 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:31.211429 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls podName:5edfb5f6-d9a0-4e21-8195-6be1703189fb nodeName:}" failed. No retries permitted until 2026-09-06 05:54:33.211413819 +0000 UTC m=+38.609744266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls") pod "image-registry-6f46956478-tn984" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb") : secret "image-registry-tls" not found Sep 06 05:54:31.449249 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:31.449161 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-trktn" event={"ID":"df6097ca-4ae8-47d6-918b-189f75f655cd","Type":"ContainerStarted","Data":"9c8d6d9cb9185052061477104dfbe60f231d2354b157785402b549e4503f299d"} Sep 06 05:54:31.452606 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:31.452575 2578 generic.go:356] "Generic (PLEG): container finished" podID="557fb632-fd3a-4b82-b25a-ba2e25072337" containerID="8ce42dbc6dbc9ebb24da625868328c8ed45d038c0811c4185b062a3dc9053445" exitCode=0 Sep 06 05:54:31.452746 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:31.452622 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4b45g" event={"ID":"557fb632-fd3a-4b82-b25a-ba2e25072337","Type":"ContainerDied","Data":"8ce42dbc6dbc9ebb24da625868328c8ed45d038c0811c4185b062a3dc9053445"} Sep 06 05:54:32.459253 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:32.459223 2578 generic.go:356] "Generic (PLEG): container finished" podID="557fb632-fd3a-4b82-b25a-ba2e25072337" containerID="14633003416f2dd48bbdd2b0affb4c4027d3e0404279bd64cf09095f1eb1aaf8" exitCode=0 Sep 06 05:54:32.459747 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:32.459283 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4b45g" event={"ID":"557fb632-fd3a-4b82-b25a-ba2e25072337","Type":"ContainerDied","Data":"14633003416f2dd48bbdd2b0affb4c4027d3e0404279bd64cf09095f1eb1aaf8"} Sep 06 05:54:33.228987 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:33.228950 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:33.229084 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:33.229055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:33.229132 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:33.229110 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:54:33.229167 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:33.229136 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f46956478-tn984: secret "image-registry-tls" not found Sep 06 05:54:33.229167 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:33.229152 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:54:33.229226 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:33.229196 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls podName:5edfb5f6-d9a0-4e21-8195-6be1703189fb nodeName:}" failed. No retries permitted until 2026-09-06 05:54:37.229176832 +0000 UTC m=+42.627507283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls") pod "image-registry-6f46956478-tn984" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb") : secret "image-registry-tls" not found Sep 06 05:54:33.229226 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:33.229210 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls podName:fee5736d-8984-42af-9121-8fe5d123a3ce nodeName:}" failed. No retries permitted until 2026-09-06 05:54:37.229204361 +0000 UTC m=+42.627534810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls") pod "dns-default-zmlqw" (UID: "fee5736d-8984-42af-9121-8fe5d123a3ce") : secret "dns-default-metrics-tls" not found Sep 06 05:54:33.463389 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:33.463296 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-trktn" event={"ID":"df6097ca-4ae8-47d6-918b-189f75f655cd","Type":"ContainerStarted","Data":"496973e6b73934be1c27f2ae75ca41786e48d9d8a55cf88fb319e069929495c7"} Sep 06 05:54:33.467369 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:33.467344 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4b45g" event={"ID":"557fb632-fd3a-4b82-b25a-ba2e25072337","Type":"ContainerStarted","Data":"0401eeac51a6c941c96df063b071b5b5fd994085c9882786b91220010b3945bc"} Sep 06 05:54:33.477144 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:33.477102 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-trktn" podStartSLOduration=1.99771907 podStartE2EDuration="4.477086451s" totalImagesPullingTime="2.479367381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:33.476355059 +0000 UTC m=+38.874685551" watchObservedRunningTime="2026-09-06 05:54:33.477086451 +0000 UTC m=+38.875416940" Sep 06 05:54:33.494269 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:33.494216 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4b45g" podStartSLOduration=9.932911475000001 podStartE2EDuration="38.494199054s" totalImagesPullingTime="28.561287579s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:54:33.49339308 +0000 UTC m=+38.891723559" watchObservedRunningTime="2026-09-06 05:54:33.494199054 +0000 UTC m=+38.892529522" Sep 06 05:54:37.259274 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:37.259227 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:37.259274 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:37.259272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:37.262381 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:37.259807 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:54:37.262381 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:37.259889 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls podName:fee5736d-8984-42af-9121-8fe5d123a3ce nodeName:}" failed. No retries permitted until 2026-09-06 05:54:45.259869874 +0000 UTC m=+50.658200335 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls") pod "dns-default-zmlqw" (UID: "fee5736d-8984-42af-9121-8fe5d123a3ce") : secret "dns-default-metrics-tls" not found Sep 06 05:54:37.262381 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:37.259718 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:54:37.262381 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:37.259954 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f46956478-tn984: secret "image-registry-tls" not found Sep 06 05:54:37.262381 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:37.260034 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls podName:5edfb5f6-d9a0-4e21-8195-6be1703189fb nodeName:}" failed. No retries permitted until 2026-09-06 05:54:45.260015164 +0000 UTC m=+50.658345633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls") pod "image-registry-6f46956478-tn984" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb") : secret "image-registry-tls" not found Sep 06 05:54:45.316535 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:45.316494 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:54:45.316535 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:45.316535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:54:45.317039 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:45.316639 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:54:45.317039 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:45.316639 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:54:45.317039 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:45.316650 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f46956478-tn984: secret "image-registry-tls" not found Sep 06 05:54:45.317039 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:45.316702 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls podName:5edfb5f6-d9a0-4e21-8195-6be1703189fb nodeName:}" failed. No retries permitted until 2026-09-06 05:55:01.316689211 +0000 UTC m=+66.715019656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls") pod "image-registry-6f46956478-tn984" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb") : secret "image-registry-tls" not found Sep 06 05:54:45.317039 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:54:45.316713 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls podName:fee5736d-8984-42af-9121-8fe5d123a3ce nodeName:}" failed. No retries permitted until 2026-09-06 05:55:01.316707651 +0000 UTC m=+66.715038096 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls") pod "dns-default-zmlqw" (UID: "fee5736d-8984-42af-9121-8fe5d123a3ce") : secret "dns-default-metrics-tls" not found Sep 06 05:54:55.442025 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:54:55.441998 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwh56" Sep 06 05:55:00.832041 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:00.832001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:55:00.832516 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:00.832051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:55:00.834296 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:00.834276 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 06 05:55:00.834343 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:00.834320 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 06 05:55:00.843214 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:55:00.843193 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 06 05:55:00.843268 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:55:00.843256 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs podName:f36acd55-4ce4-4bdb-9518-cca261651a19 nodeName:}" failed. No retries permitted until 2026-09-06 05:56:04.843238201 +0000 UTC m=+130.241568646 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs") pod "network-metrics-daemon-w74pq" (UID: "f36acd55-4ce4-4bdb-9518-cca261651a19") : secret "metrics-daemon-secret" not found Sep 06 05:55:00.846679 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:00.846642 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/115882c1-6556-4feb-adc9-a5ddf5f36ba2-original-pull-secret\") pod \"global-pull-secret-syncer-c66ws\" (UID: \"115882c1-6556-4feb-adc9-a5ddf5f36ba2\") " pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:55:00.859985 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:00.859959 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c66ws" Sep 06 05:55:01.034115 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:01.034070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bn8x\" (UniqueName: \"kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x\") pod \"network-check-target-f87xg\" (UID: \"a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40\") " pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:55:01.036434 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:01.036417 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 06 05:55:01.046999 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:01.046983 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 06 05:55:01.058195 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:01.058169 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bn8x\" (UniqueName: \"kubernetes.io/projected/a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40-kube-api-access-7bn8x\") pod \"network-check-target-f87xg\" (UID: \"a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40\") " pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:55:01.151554 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:01.151489 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n9n5z\"" Sep 06 05:55:01.160176 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:01.160148 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c66ws"] Sep 06 05:55:01.160324 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:01.160178 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:55:01.163302 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:55:01.163268 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115882c1_6556_4feb_adc9_a5ddf5f36ba2.slice/crio-bbc4bc3b43b0f1b3144d178cc72879a944fd1d40fc537b117062795fdd3570ea WatchSource:0}: Error finding container bbc4bc3b43b0f1b3144d178cc72879a944fd1d40fc537b117062795fdd3570ea: Status 404 returned error can't find the container with id bbc4bc3b43b0f1b3144d178cc72879a944fd1d40fc537b117062795fdd3570ea Sep 06 05:55:01.336915 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:01.336876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:55:01.337060 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:01.336927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:55:01.337060 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:55:01.337030 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:55:01.337060 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:55:01.337053 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:55:01.337176 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:55:01.337064 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f46956478-tn984: secret "image-registry-tls" not found Sep 06 05:55:01.337176 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:55:01.337097 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls podName:fee5736d-8984-42af-9121-8fe5d123a3ce nodeName:}" failed. No retries permitted until 2026-09-06 05:55:33.337081013 +0000 UTC m=+98.735411476 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls") pod "dns-default-zmlqw" (UID: "fee5736d-8984-42af-9121-8fe5d123a3ce") : secret "dns-default-metrics-tls" not found Sep 06 05:55:01.337176 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:55:01.337113 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls podName:5edfb5f6-d9a0-4e21-8195-6be1703189fb nodeName:}" failed. No retries permitted until 2026-09-06 05:55:33.337106932 +0000 UTC m=+98.735437396 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls") pod "image-registry-6f46956478-tn984" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb") : secret "image-registry-tls" not found Sep 06 05:55:01.443964 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:01.443821 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f87xg"] Sep 06 05:55:01.447392 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:55:01.447360 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29d8c30_e987_4ec5_85b0_6a9ad1ef1a40.slice/crio-dcf70ed3a27fc545c44701119cf35d79cb4765f599710ac550f50ffa764eaf9c WatchSource:0}: Error finding container dcf70ed3a27fc545c44701119cf35d79cb4765f599710ac550f50ffa764eaf9c: Status 404 returned error can't find the container with id dcf70ed3a27fc545c44701119cf35d79cb4765f599710ac550f50ffa764eaf9c Sep 06 05:55:01.533981 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:01.533940 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c66ws" event={"ID":"115882c1-6556-4feb-adc9-a5ddf5f36ba2","Type":"ContainerStarted","Data":"bbc4bc3b43b0f1b3144d178cc72879a944fd1d40fc537b117062795fdd3570ea"} Sep 06 05:55:01.535010 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:01.534978 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f87xg" event={"ID":"a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40","Type":"ContainerStarted","Data":"dcf70ed3a27fc545c44701119cf35d79cb4765f599710ac550f50ffa764eaf9c"} Sep 06 05:55:06.549807 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:06.549757 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c66ws" event={"ID":"115882c1-6556-4feb-adc9-a5ddf5f36ba2","Type":"ContainerStarted","Data":"1038f2a0539823124b10dac332805930c28dc730eca8758a2abbbd5117076dff"} Sep 06 05:55:06.551237 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:06.551213 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f87xg" event={"ID":"a29d8c30-e987-4ec5-85b0-6a9ad1ef1a40","Type":"ContainerStarted","Data":"ad7b50b2cf9a51939f8d9f040281d52bd12e49499cf9000b302ccc96971e34c8"} Sep 06 05:55:06.551379 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:06.551350 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:55:06.563017 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:06.562974 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c66ws" podStartSLOduration=66.90654983 podStartE2EDuration="1m11.562961083s" totalImagesPullingTime="4.656411253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:55:06.562642135 +0000 UTC m=+71.960972603" watchObservedRunningTime="2026-09-06 05:55:06.562961083 +0000 UTC m=+71.961291550" Sep 06 05:55:06.575534 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:06.575492 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f87xg" podStartSLOduration=67.209100707 podStartE2EDuration="1m11.575478014s" totalImagesPullingTime="4.366377307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:55:06.574797046 +0000 UTC m=+71.973127514" watchObservedRunningTime="2026-09-06 05:55:06.575478014 +0000 UTC m=+71.973808480" Sep 06 05:55:33.370490 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:33.370410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:55:33.370490 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:33.370455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:55:33.370977 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:55:33.370554 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:55:33.370977 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:55:33.370558 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:55:33.370977 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:55:33.370628 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls podName:fee5736d-8984-42af-9121-8fe5d123a3ce nodeName:}" failed. No retries permitted until 2026-09-06 05:56:37.370613396 +0000 UTC m=+162.768943841 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls") pod "dns-default-zmlqw" (UID: "fee5736d-8984-42af-9121-8fe5d123a3ce") : secret "dns-default-metrics-tls" not found Sep 06 05:55:33.370977 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:55:33.370565 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f46956478-tn984: secret "image-registry-tls" not found Sep 06 05:55:33.370977 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:55:33.370697 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls podName:5edfb5f6-d9a0-4e21-8195-6be1703189fb nodeName:}" failed. No retries permitted until 2026-09-06 05:56:37.370686533 +0000 UTC m=+162.769016978 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls") pod "image-registry-6f46956478-tn984" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb") : secret "image-registry-tls" not found Sep 06 05:55:34.887414 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:34.887380 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8"] Sep 06 05:55:34.891585 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:34.891566 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8" Sep 06 05:55:34.894780 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:34.894748 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 06 05:55:34.894927 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:34.894748 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 06 05:55:34.894927 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:34.894793 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 06 05:55:34.894927 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:34.894849 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 06 05:55:34.895106 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:34.895072 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-vs78h\"" Sep 06 05:55:34.900205 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:34.900186 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8"] Sep 06 05:55:34.983517 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:34.983483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5cdh\" (UniqueName: \"kubernetes.io/projected/ec0ed034-7e9d-4642-88db-fd1a37d767a7-kube-api-access-j5cdh\") pod \"managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8\" (UID: \"ec0ed034-7e9d-4642-88db-fd1a37d767a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8" Sep 06 05:55:34.983517 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:34.983526 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ec0ed034-7e9d-4642-88db-fd1a37d767a7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8\" (UID: \"ec0ed034-7e9d-4642-88db-fd1a37d767a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8" Sep 06 05:55:35.084711 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:35.084679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5cdh\" (UniqueName: \"kubernetes.io/projected/ec0ed034-7e9d-4642-88db-fd1a37d767a7-kube-api-access-j5cdh\") pod \"managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8\" (UID: \"ec0ed034-7e9d-4642-88db-fd1a37d767a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8" Sep 06 05:55:35.084817 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:35.084721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ec0ed034-7e9d-4642-88db-fd1a37d767a7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8\" (UID: \"ec0ed034-7e9d-4642-88db-fd1a37d767a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8" Sep 06 05:55:35.087382 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:35.087356 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ec0ed034-7e9d-4642-88db-fd1a37d767a7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8\" (UID: \"ec0ed034-7e9d-4642-88db-fd1a37d767a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8" Sep 06 05:55:35.092005 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:35.091975 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5cdh\" (UniqueName: \"kubernetes.io/projected/ec0ed034-7e9d-4642-88db-fd1a37d767a7-kube-api-access-j5cdh\") pod \"managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8\" (UID: \"ec0ed034-7e9d-4642-88db-fd1a37d767a7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8" Sep 06 05:55:35.211316 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:35.211208 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8" Sep 06 05:55:35.476519 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:35.476436 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8"] Sep 06 05:55:35.480633 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:55:35.480601 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec0ed034_7e9d_4642_88db_fd1a37d767a7.slice/crio-20cc334dc30d06c4daca099dd04814fa4b7b86776c1e22c9b24552ec82d36e85 WatchSource:0}: Error finding container 20cc334dc30d06c4daca099dd04814fa4b7b86776c1e22c9b24552ec82d36e85: Status 404 returned error can't find the container with id 20cc334dc30d06c4daca099dd04814fa4b7b86776c1e22c9b24552ec82d36e85 Sep 06 05:55:35.627808 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:35.627771 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8" event={"ID":"ec0ed034-7e9d-4642-88db-fd1a37d767a7","Type":"ContainerStarted","Data":"20cc334dc30d06c4daca099dd04814fa4b7b86776c1e22c9b24552ec82d36e85"} Sep 06 05:55:37.555696 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:37.555667 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f87xg" Sep 06 05:55:38.637227 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:38.637189 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8" event={"ID":"ec0ed034-7e9d-4642-88db-fd1a37d767a7","Type":"ContainerStarted","Data":"d4eca8986378950a0510aacfffb3b595fbbdd40062dd0bcb06650627d6c52152"} Sep 06 05:55:38.651590 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:55:38.651542 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8" podStartSLOduration=2.082061709 podStartE2EDuration="4.651527236s" totalImagesPullingTime="2.569465527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:55:38.650976557 +0000 UTC m=+104.049307026" watchObservedRunningTime="2026-09-06 05:55:38.651527236 +0000 UTC m=+104.049857754" Sep 06 05:56:04.899371 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:56:04.899329 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:56:04.899868 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:56:04.899503 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 06 05:56:04.899868 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:56:04.899603 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs podName:f36acd55-4ce4-4bdb-9518-cca261651a19 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:06.899578281 +0000 UTC m=+252.297908727 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs") pod "network-metrics-daemon-w74pq" (UID: "f36acd55-4ce4-4bdb-9518-cca261651a19") : secret "metrics-daemon-secret" not found Sep 06 05:56:32.455968 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:56:32.455929 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6f46956478-tn984" podUID="5edfb5f6-d9a0-4e21-8195-6be1703189fb" Sep 06 05:56:32.464128 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:56:32.464096 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zmlqw" podUID="fee5736d-8984-42af-9121-8fe5d123a3ce" Sep 06 05:56:32.772217 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:56:32.772192 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmlqw" Sep 06 05:56:32.772353 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:56:32.772192 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:56:34.154831 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:56:34.154774 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:56:37.432162 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:56:37.432102 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:56:37.432162 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:56:37.432163 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:56:37.432676 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:56:37.432261 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 06 05:56:37.432676 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:56:37.432321 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 06 05:56:37.432676 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:56:37.432338 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f46956478-tn984: secret "image-registry-tls" not found Sep 06 05:56:37.432676 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:56:37.432326 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls podName:fee5736d-8984-42af-9121-8fe5d123a3ce nodeName:}" failed. No retries permitted until 2026-09-06 05:58:39.432309895 +0000 UTC m=+284.830640342 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls") pod "dns-default-zmlqw" (UID: "fee5736d-8984-42af-9121-8fe5d123a3ce") : secret "dns-default-metrics-tls" not found Sep 06 05:56:37.432676 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:56:37.432407 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls podName:5edfb5f6-d9a0-4e21-8195-6be1703189fb nodeName:}" failed. No retries permitted until 2026-09-06 05:58:39.432390205 +0000 UTC m=+284.830720652 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls") pod "image-registry-6f46956478-tn984" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb") : secret "image-registry-tls" not found Sep 06 05:56:49.138671 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:56:49.138571 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:57:38.932755 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:57:38.932723 2578 generic.go:356] "Generic (PLEG): container finished" podID="ec0ed034-7e9d-4642-88db-fd1a37d767a7" containerID="d4eca8986378950a0510aacfffb3b595fbbdd40062dd0bcb06650627d6c52152" exitCode=255 Sep 06 05:57:38.933192 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:57:38.932761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8" event={"ID":"ec0ed034-7e9d-4642-88db-fd1a37d767a7","Type":"ContainerDied","Data":"d4eca8986378950a0510aacfffb3b595fbbdd40062dd0bcb06650627d6c52152"} Sep 06 05:57:38.938282 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:57:38.938263 2578 scope.go:117] "RemoveContainer" containerID="d4eca8986378950a0510aacfffb3b595fbbdd40062dd0bcb06650627d6c52152" Sep 06 05:57:39.937053 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:57:39.937016 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fff6cf4c8-ntsq8" event={"ID":"ec0ed034-7e9d-4642-88db-fd1a37d767a7","Type":"ContainerStarted","Data":"5d7cc3916b388dce449660d4fc7fdc49c887df607d07bfa936802a48eb73f9e6"} Sep 06 05:58:06.965464 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:06.965422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:58:06.965829 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:06.965585 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 06 05:58:06.965829 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:06.965655 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs podName:f36acd55-4ce4-4bdb-9518-cca261651a19 nodeName:}" failed. No retries permitted until 2026-09-06 06:00:08.965640055 +0000 UTC m=+374.363970504 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs") pod "network-metrics-daemon-w74pq" (UID: "f36acd55-4ce4-4bdb-9518-cca261651a19") : secret "metrics-daemon-secret" not found Sep 06 05:58:22.668737 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.668662 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-99f956f9b-9d6ht"] Sep 06 05:58:22.671467 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.671448 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:22.673787 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.673753 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 06 05:58:22.673787 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.673765 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8qfrf\"" Sep 06 05:58:22.673787 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.673772 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 06 05:58:22.674060 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.673800 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 06 05:58:22.674060 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.674038 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 06 05:58:22.674168 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.674113 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 06 05:58:22.674424 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.674409 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 06 05:58:22.685189 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.685162 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-99f956f9b-9d6ht"] Sep 06 05:58:22.758885 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.758850 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-5zkxr"] Sep 06 05:58:22.761834 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.761767 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-5zkxr" Sep 06 05:58:22.762769 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.762750 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8x82j"] Sep 06 05:58:22.764149 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.764127 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-52grw\"" Sep 06 05:58:22.765446 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.765424 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-6jcbw"] Sep 06 05:58:22.765548 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.765534 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:22.767930 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.767711 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 06 05:58:22.767930 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.767817 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 06 05:58:22.768098 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.767987 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 06 05:58:22.768369 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.768153 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 06 05:58:22.768369 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.768214 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-n8rsp\"" Sep 06 05:58:22.768586 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.768566 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:22.768713 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.768695 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-stats-auth\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:22.768833 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.768816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzlvv\" (UniqueName: \"kubernetes.io/projected/c9644473-eedd-4380-a23d-ecf5b0c85f10-kube-api-access-lzlvv\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:22.769037 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.768958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:22.769037 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.769002 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-default-certificate\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:22.770335 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.770279 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt"] Sep 06 05:58:22.770618 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.770602 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.774096 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.774079 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 06 05:58:22.774199 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.774090 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 06 05:58:22.774290 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.774119 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 06 05:58:22.774349 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.774129 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 06 05:58:22.774823 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.774805 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-f5xmh\"" Sep 06 05:58:22.775618 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.775592 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-5zkxr"] Sep 06 05:58:22.775926 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.775741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" Sep 06 05:58:22.776450 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.776434 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 06 05:58:22.778870 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.778750 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt"] Sep 06 05:58:22.779149 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.779128 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 06 05:58:22.779369 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.779350 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 06 05:58:22.780017 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.779996 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7n64t\"" Sep 06 05:58:22.780187 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.780169 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 06 05:58:22.780252 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.780205 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-6jcbw"] Sep 06 05:58:22.780308 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.780283 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 06 05:58:22.780996 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.780976 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 06 05:58:22.781350 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.781333 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8x82j"] Sep 06 05:58:22.869875 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.869837 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50fa393c-a0bf-4082-8040-8018f2ba260c-serving-cert\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.869875 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.869880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-default-certificate\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:22.870125 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.869919 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c23eec1b-1145-4e2d-9d9f-4b76157f40a0-config\") pod \"console-operator-85b454874b-8x82j\" (UID: \"c23eec1b-1145-4e2d-9d9f-4b76157f40a0\") " pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:22.870125 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.869957 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c23eec1b-1145-4e2d-9d9f-4b76157f40a0-trusted-ca\") pod \"console-operator-85b454874b-8x82j\" (UID: \"c23eec1b-1145-4e2d-9d9f-4b76157f40a0\") " pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:22.870125 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.870050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7br2\" (UniqueName: \"kubernetes.io/projected/c23eec1b-1145-4e2d-9d9f-4b76157f40a0-kube-api-access-h7br2\") pod \"console-operator-85b454874b-8x82j\" (UID: \"c23eec1b-1145-4e2d-9d9f-4b76157f40a0\") " pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:22.870125 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.870099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:22.870125 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.870120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-stats-auth\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:22.870366 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.870135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c23eec1b-1145-4e2d-9d9f-4b76157f40a0-serving-cert\") pod \"console-operator-85b454874b-8x82j\" (UID: \"c23eec1b-1145-4e2d-9d9f-4b76157f40a0\") " pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:22.870366 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.870153 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50fa393c-a0bf-4082-8040-8018f2ba260c-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.870366 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.870204 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74d86613-a3ce-4adf-bf86-934052abad09-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ff6vt\" (UID: \"74d86613-a3ce-4adf-bf86-934052abad09\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" Sep 06 05:58:22.870366 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.870244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50fa393c-a0bf-4082-8040-8018f2ba260c-tmp\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.870366 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:22.870287 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle podName:c9644473-eedd-4380-a23d-ecf5b0c85f10 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:23.370261664 +0000 UTC m=+268.768592110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle") pod "router-default-99f956f9b-9d6ht" (UID: "c9644473-eedd-4380-a23d-ecf5b0c85f10") : configmap references non-existent config key: service-ca.crt Sep 06 05:58:22.870366 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.870329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/50fa393c-a0bf-4082-8040-8018f2ba260c-snapshots\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.870366 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.870348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74d86613-a3ce-4adf-bf86-934052abad09-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ff6vt\" (UID: \"74d86613-a3ce-4adf-bf86-934052abad09\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" Sep 06 05:58:22.870695 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.870407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7ctx\" (UniqueName: \"kubernetes.io/projected/50fa393c-a0bf-4082-8040-8018f2ba260c-kube-api-access-k7ctx\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.870695 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.870460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndd8v\" (UniqueName: \"kubernetes.io/projected/74d86613-a3ce-4adf-bf86-934052abad09-kube-api-access-ndd8v\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ff6vt\" (UID: \"74d86613-a3ce-4adf-bf86-934052abad09\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" Sep 06 05:58:22.870695 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.870493 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qz9p\" (UniqueName: \"kubernetes.io/projected/61af572d-2757-4ac9-9f8f-687c2aba6f53-kube-api-access-8qz9p\") pod \"network-check-source-685456655f-5zkxr\" (UID: \"61af572d-2757-4ac9-9f8f-687c2aba6f53\") " pod="openshift-network-diagnostics/network-check-source-685456655f-5zkxr" Sep 06 05:58:22.870695 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.870524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzlvv\" (UniqueName: \"kubernetes.io/projected/c9644473-eedd-4380-a23d-ecf5b0c85f10-kube-api-access-lzlvv\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:22.870695 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.870543 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50fa393c-a0bf-4082-8040-8018f2ba260c-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.870695 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.870566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:22.870695 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:22.870645 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 06 05:58:22.871022 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:22.870747 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs podName:c9644473-eedd-4380-a23d-ecf5b0c85f10 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:23.370700336 +0000 UTC m=+268.769030783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs") pod "router-default-99f956f9b-9d6ht" (UID: "c9644473-eedd-4380-a23d-ecf5b0c85f10") : secret "router-metrics-certs-default" not found Sep 06 05:58:22.872543 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.872520 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-default-certificate\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:22.872543 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.872542 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-stats-auth\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:22.878482 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.878454 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzlvv\" (UniqueName: \"kubernetes.io/projected/c9644473-eedd-4380-a23d-ecf5b0c85f10-kube-api-access-lzlvv\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:22.971837 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.971752 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/50fa393c-a0bf-4082-8040-8018f2ba260c-snapshots\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.971837 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.971792 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74d86613-a3ce-4adf-bf86-934052abad09-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ff6vt\" (UID: \"74d86613-a3ce-4adf-bf86-934052abad09\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" Sep 06 05:58:22.971837 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.971812 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7ctx\" (UniqueName: \"kubernetes.io/projected/50fa393c-a0bf-4082-8040-8018f2ba260c-kube-api-access-k7ctx\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.972144 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.971861 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndd8v\" (UniqueName: \"kubernetes.io/projected/74d86613-a3ce-4adf-bf86-934052abad09-kube-api-access-ndd8v\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ff6vt\" (UID: \"74d86613-a3ce-4adf-bf86-934052abad09\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" Sep 06 05:58:22.972144 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.971894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qz9p\" (UniqueName: \"kubernetes.io/projected/61af572d-2757-4ac9-9f8f-687c2aba6f53-kube-api-access-8qz9p\") pod \"network-check-source-685456655f-5zkxr\" (UID: \"61af572d-2757-4ac9-9f8f-687c2aba6f53\") " pod="openshift-network-diagnostics/network-check-source-685456655f-5zkxr" Sep 06 05:58:22.972144 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.971940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50fa393c-a0bf-4082-8040-8018f2ba260c-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.972144 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.971984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50fa393c-a0bf-4082-8040-8018f2ba260c-serving-cert\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.972144 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.972007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c23eec1b-1145-4e2d-9d9f-4b76157f40a0-config\") pod \"console-operator-85b454874b-8x82j\" (UID: \"c23eec1b-1145-4e2d-9d9f-4b76157f40a0\") " pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:22.972144 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.972032 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c23eec1b-1145-4e2d-9d9f-4b76157f40a0-trusted-ca\") pod \"console-operator-85b454874b-8x82j\" (UID: \"c23eec1b-1145-4e2d-9d9f-4b76157f40a0\") " pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:22.972144 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.972063 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7br2\" (UniqueName: \"kubernetes.io/projected/c23eec1b-1145-4e2d-9d9f-4b76157f40a0-kube-api-access-h7br2\") pod \"console-operator-85b454874b-8x82j\" (UID: \"c23eec1b-1145-4e2d-9d9f-4b76157f40a0\") " pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:22.972462 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.972166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c23eec1b-1145-4e2d-9d9f-4b76157f40a0-serving-cert\") pod \"console-operator-85b454874b-8x82j\" (UID: \"c23eec1b-1145-4e2d-9d9f-4b76157f40a0\") " pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:22.972462 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.972201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50fa393c-a0bf-4082-8040-8018f2ba260c-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.972462 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.972237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74d86613-a3ce-4adf-bf86-934052abad09-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ff6vt\" (UID: \"74d86613-a3ce-4adf-bf86-934052abad09\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" Sep 06 05:58:22.972462 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.972272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50fa393c-a0bf-4082-8040-8018f2ba260c-tmp\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.972657 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.972492 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74d86613-a3ce-4adf-bf86-934052abad09-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ff6vt\" (UID: \"74d86613-a3ce-4adf-bf86-934052abad09\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" Sep 06 05:58:22.972657 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.972624 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/50fa393c-a0bf-4082-8040-8018f2ba260c-snapshots\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.972774 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.972675 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50fa393c-a0bf-4082-8040-8018f2ba260c-tmp\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.972874 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.972846 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c23eec1b-1145-4e2d-9d9f-4b76157f40a0-config\") pod \"console-operator-85b454874b-8x82j\" (UID: \"c23eec1b-1145-4e2d-9d9f-4b76157f40a0\") " pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:22.973005 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.972986 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50fa393c-a0bf-4082-8040-8018f2ba260c-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.973060 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.973023 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50fa393c-a0bf-4082-8040-8018f2ba260c-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.973758 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.973733 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c23eec1b-1145-4e2d-9d9f-4b76157f40a0-trusted-ca\") pod \"console-operator-85b454874b-8x82j\" (UID: \"c23eec1b-1145-4e2d-9d9f-4b76157f40a0\") " pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:22.974740 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.974721 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c23eec1b-1145-4e2d-9d9f-4b76157f40a0-serving-cert\") pod \"console-operator-85b454874b-8x82j\" (UID: \"c23eec1b-1145-4e2d-9d9f-4b76157f40a0\") " pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:22.974875 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.974857 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50fa393c-a0bf-4082-8040-8018f2ba260c-serving-cert\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.974958 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.974897 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74d86613-a3ce-4adf-bf86-934052abad09-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ff6vt\" (UID: \"74d86613-a3ce-4adf-bf86-934052abad09\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" Sep 06 05:58:22.979921 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.979883 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7br2\" (UniqueName: \"kubernetes.io/projected/c23eec1b-1145-4e2d-9d9f-4b76157f40a0-kube-api-access-h7br2\") pod \"console-operator-85b454874b-8x82j\" (UID: \"c23eec1b-1145-4e2d-9d9f-4b76157f40a0\") " pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:22.980448 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.980424 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7ctx\" (UniqueName: \"kubernetes.io/projected/50fa393c-a0bf-4082-8040-8018f2ba260c-kube-api-access-k7ctx\") pod \"insights-operator-76bcd6ccf5-6jcbw\" (UID: \"50fa393c-a0bf-4082-8040-8018f2ba260c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:22.980599 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.980537 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndd8v\" (UniqueName: \"kubernetes.io/projected/74d86613-a3ce-4adf-bf86-934052abad09-kube-api-access-ndd8v\") pod \"kube-storage-version-migrator-operator-8d7dd4675-ff6vt\" (UID: \"74d86613-a3ce-4adf-bf86-934052abad09\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" Sep 06 05:58:22.980992 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:22.980973 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qz9p\" (UniqueName: \"kubernetes.io/projected/61af572d-2757-4ac9-9f8f-687c2aba6f53-kube-api-access-8qz9p\") pod \"network-check-source-685456655f-5zkxr\" (UID: \"61af572d-2757-4ac9-9f8f-687c2aba6f53\") " pod="openshift-network-diagnostics/network-check-source-685456655f-5zkxr" Sep 06 05:58:23.074009 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:23.073971 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-5zkxr" Sep 06 05:58:23.083190 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:23.083162 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:23.089086 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:23.089060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" Sep 06 05:58:23.093979 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:23.093952 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" Sep 06 05:58:23.353972 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:23.353943 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-5zkxr"] Sep 06 05:58:23.356979 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:58:23.356949 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61af572d_2757_4ac9_9f8f_687c2aba6f53.slice/crio-45f724416c696dcc9d3df6d557210900e4843012d7e98ea5a7e94883729018ab WatchSource:0}: Error finding container 45f724416c696dcc9d3df6d557210900e4843012d7e98ea5a7e94883729018ab: Status 404 returned error can't find the container with id 45f724416c696dcc9d3df6d557210900e4843012d7e98ea5a7e94883729018ab Sep 06 05:58:23.375223 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:23.375200 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:23.375331 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:23.375257 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:23.375423 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:23.375405 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle podName:c9644473-eedd-4380-a23d-ecf5b0c85f10 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:24.37538479 +0000 UTC m=+269.773715241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle") pod "router-default-99f956f9b-9d6ht" (UID: "c9644473-eedd-4380-a23d-ecf5b0c85f10") : configmap references non-existent config key: service-ca.crt Sep 06 05:58:23.376047 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:23.375427 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 06 05:58:23.376047 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:23.375491 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs podName:c9644473-eedd-4380-a23d-ecf5b0c85f10 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:24.37547063 +0000 UTC m=+269.773801078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs") pod "router-default-99f956f9b-9d6ht" (UID: "c9644473-eedd-4380-a23d-ecf5b0c85f10") : secret "router-metrics-certs-default" not found Sep 06 05:58:23.376143 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:23.376104 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8x82j"] Sep 06 05:58:23.383616 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:23.383251 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-6jcbw"] Sep 06 05:58:23.383982 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:58:23.383957 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23eec1b_1145_4e2d_9d9f_4b76157f40a0.slice/crio-02ec9ba29aae7b6a6c9489458eb2188153e7398b5d0949212f80cc15ecddfbe2 WatchSource:0}: Error finding container 02ec9ba29aae7b6a6c9489458eb2188153e7398b5d0949212f80cc15ecddfbe2: Status 404 returned error can't find the container with id 02ec9ba29aae7b6a6c9489458eb2188153e7398b5d0949212f80cc15ecddfbe2 Sep 06 05:58:23.385975 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:58:23.385730 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fa393c_a0bf_4082_8040_8018f2ba260c.slice/crio-272fcf4eb45bab11f280d29d0fda5a3d780dfc86d0c12e8610ed30f9398946e4 WatchSource:0}: Error finding container 272fcf4eb45bab11f280d29d0fda5a3d780dfc86d0c12e8610ed30f9398946e4: Status 404 returned error can't find the container with id 272fcf4eb45bab11f280d29d0fda5a3d780dfc86d0c12e8610ed30f9398946e4 Sep 06 05:58:23.408537 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:23.408514 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt"] Sep 06 05:58:23.411309 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:58:23.411277 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d86613_a3ce_4adf_bf86_934052abad09.slice/crio-97b84b292df71870e2653b854d57f8d4853d687ee36d2248f744b4d53d492e6f WatchSource:0}: Error finding container 97b84b292df71870e2653b854d57f8d4853d687ee36d2248f744b4d53d492e6f: Status 404 returned error can't find the container with id 97b84b292df71870e2653b854d57f8d4853d687ee36d2248f744b4d53d492e6f Sep 06 05:58:24.053430 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:24.053385 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8x82j" event={"ID":"c23eec1b-1145-4e2d-9d9f-4b76157f40a0","Type":"ContainerStarted","Data":"02ec9ba29aae7b6a6c9489458eb2188153e7398b5d0949212f80cc15ecddfbe2"} Sep 06 05:58:24.056086 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:24.055795 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" event={"ID":"74d86613-a3ce-4adf-bf86-934052abad09","Type":"ContainerStarted","Data":"97b84b292df71870e2653b854d57f8d4853d687ee36d2248f744b4d53d492e6f"} Sep 06 05:58:24.058205 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:24.058170 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" event={"ID":"50fa393c-a0bf-4082-8040-8018f2ba260c","Type":"ContainerStarted","Data":"272fcf4eb45bab11f280d29d0fda5a3d780dfc86d0c12e8610ed30f9398946e4"} Sep 06 05:58:24.061615 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:24.061416 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-5zkxr" event={"ID":"61af572d-2757-4ac9-9f8f-687c2aba6f53","Type":"ContainerStarted","Data":"30ed3d5e7bf8aaf0a57f5517efcf206f7fe75b1a313204f43eee3e45b9c111d8"} Sep 06 05:58:24.061615 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:24.061457 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-5zkxr" event={"ID":"61af572d-2757-4ac9-9f8f-687c2aba6f53","Type":"ContainerStarted","Data":"45f724416c696dcc9d3df6d557210900e4843012d7e98ea5a7e94883729018ab"} Sep 06 05:58:24.383500 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:24.383415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:24.383678 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:24.383528 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:24.383678 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:24.383653 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 06 05:58:24.383803 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:24.383712 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs podName:c9644473-eedd-4380-a23d-ecf5b0c85f10 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:26.383694356 +0000 UTC m=+271.782024809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs") pod "router-default-99f956f9b-9d6ht" (UID: "c9644473-eedd-4380-a23d-ecf5b0c85f10") : secret "router-metrics-certs-default" not found Sep 06 05:58:24.384132 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:24.384108 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle podName:c9644473-eedd-4380-a23d-ecf5b0c85f10 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:26.384085765 +0000 UTC m=+271.782416232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle") pod "router-default-99f956f9b-9d6ht" (UID: "c9644473-eedd-4380-a23d-ecf5b0c85f10") : configmap references non-existent config key: service-ca.crt Sep 06 05:58:26.401372 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:26.401337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:26.401853 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:26.401444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:26.401853 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:26.401536 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle podName:c9644473-eedd-4380-a23d-ecf5b0c85f10 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:30.401512838 +0000 UTC m=+275.799843301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle") pod "router-default-99f956f9b-9d6ht" (UID: "c9644473-eedd-4380-a23d-ecf5b0c85f10") : configmap references non-existent config key: service-ca.crt Sep 06 05:58:26.401853 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:26.401574 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 06 05:58:26.401853 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:26.401633 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs podName:c9644473-eedd-4380-a23d-ecf5b0c85f10 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:30.401615299 +0000 UTC m=+275.799945758 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs") pod "router-default-99f956f9b-9d6ht" (UID: "c9644473-eedd-4380-a23d-ecf5b0c85f10") : secret "router-metrics-certs-default" not found Sep 06 05:58:27.073979 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.073949 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/0.log" Sep 06 05:58:27.074164 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.073994 2578 generic.go:356] "Generic (PLEG): container finished" podID="c23eec1b-1145-4e2d-9d9f-4b76157f40a0" containerID="4bdf3255d5dde32e3351f54d77028557a5dffee156b5074b920c035327d5126f" exitCode=255 Sep 06 05:58:27.074164 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.074096 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8x82j" event={"ID":"c23eec1b-1145-4e2d-9d9f-4b76157f40a0","Type":"ContainerDied","Data":"4bdf3255d5dde32e3351f54d77028557a5dffee156b5074b920c035327d5126f"} Sep 06 05:58:27.074335 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.074319 2578 scope.go:117] "RemoveContainer" containerID="4bdf3255d5dde32e3351f54d77028557a5dffee156b5074b920c035327d5126f" Sep 06 05:58:27.075860 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.075831 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" event={"ID":"74d86613-a3ce-4adf-bf86-934052abad09","Type":"ContainerStarted","Data":"187cc5b48080ec93b96673e51755e161ed8cc9cdc754c5580a00af09fc43a9b2"} Sep 06 05:58:27.077461 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.077432 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" event={"ID":"50fa393c-a0bf-4082-8040-8018f2ba260c","Type":"ContainerStarted","Data":"e234271a8c3574385afb3a6aed5e60f0d6e0e64ef480f5baf1b3ec8496268266"} Sep 06 05:58:27.089418 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.089373 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-5zkxr" podStartSLOduration=5.08935386 podStartE2EDuration="5.08935386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:24.076674922 +0000 UTC m=+269.475005416" watchObservedRunningTime="2026-09-06 05:58:27.08935386 +0000 UTC m=+272.487684327" Sep 06 05:58:27.103095 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.103045 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" podStartSLOduration=1.948900786 podStartE2EDuration="5.103027136s" totalImagesPullingTime="3.15412635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:27.102257124 +0000 UTC m=+272.500587614" watchObservedRunningTime="2026-09-06 05:58:27.103027136 +0000 UTC m=+272.501357608" Sep 06 05:58:27.116089 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.116042 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" podStartSLOduration=1.979910917 podStartE2EDuration="5.116021602s" totalImagesPullingTime="3.136110685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:27.115062222 +0000 UTC m=+272.513392701" watchObservedRunningTime="2026-09-06 05:58:27.116021602 +0000 UTC m=+272.514352073" Sep 06 05:58:27.530832 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.530794 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-llpf2"] Sep 06 05:58:27.533916 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.533886 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:27.536279 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.536255 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 06 05:58:27.536368 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.536256 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5nmlz\"" Sep 06 05:58:27.537127 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.537111 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 06 05:58:27.546324 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.546297 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-llpf2"] Sep 06 05:58:27.612069 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.612033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c05aa74a-9686-4a1c-a26e-68ef0f755859-data-volume\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:27.612199 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.612079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:27.612199 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.612107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:27.612199 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.612185 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c05aa74a-9686-4a1c-a26e-68ef0f755859-crio-socket\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:27.612311 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.612260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snnz7\" (UniqueName: \"kubernetes.io/projected/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-api-access-snnz7\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:27.713616 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.713580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snnz7\" (UniqueName: \"kubernetes.io/projected/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-api-access-snnz7\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:27.713789 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.713646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c05aa74a-9686-4a1c-a26e-68ef0f755859-data-volume\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:27.713789 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.713690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:27.713789 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.713717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:27.713789 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.713775 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c05aa74a-9686-4a1c-a26e-68ef0f755859-crio-socket\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:27.714040 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:27.713801 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 06 05:58:27.714040 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:27.713843 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 06 05:58:27.714040 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.713873 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c05aa74a-9686-4a1c-a26e-68ef0f755859-crio-socket\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:27.714040 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:27.713875 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm podName:c05aa74a-9686-4a1c-a26e-68ef0f755859 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:28.213855137 +0000 UTC m=+273.612185583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llpf2" (UID: "c05aa74a-9686-4a1c-a26e-68ef0f755859") : configmap "kube-rbac-proxy" not found Sep 06 05:58:27.714040 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:27.713949 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls podName:c05aa74a-9686-4a1c-a26e-68ef0f755859 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:28.213929748 +0000 UTC m=+273.612260204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llpf2" (UID: "c05aa74a-9686-4a1c-a26e-68ef0f755859") : secret "insights-runtime-extractor-tls" not found Sep 06 05:58:27.714294 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.714180 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c05aa74a-9686-4a1c-a26e-68ef0f755859-data-volume\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:27.726947 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:27.726920 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snnz7\" (UniqueName: \"kubernetes.io/projected/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-api-access-snnz7\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:28.082212 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:28.082183 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/1.log" Sep 06 05:58:28.082638 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:28.082623 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/0.log" Sep 06 05:58:28.082704 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:28.082657 2578 generic.go:356] "Generic (PLEG): container finished" podID="c23eec1b-1145-4e2d-9d9f-4b76157f40a0" containerID="37b8933f23d64cdbb28a3cf90a64676f122498ae99c76976f8067863ab00fcf3" exitCode=255 Sep 06 05:58:28.082759 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:28.082714 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8x82j" event={"ID":"c23eec1b-1145-4e2d-9d9f-4b76157f40a0","Type":"ContainerDied","Data":"37b8933f23d64cdbb28a3cf90a64676f122498ae99c76976f8067863ab00fcf3"} Sep 06 05:58:28.082812 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:28.082775 2578 scope.go:117] "RemoveContainer" containerID="4bdf3255d5dde32e3351f54d77028557a5dffee156b5074b920c035327d5126f" Sep 06 05:58:28.082959 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:28.082945 2578 scope.go:117] "RemoveContainer" containerID="37b8933f23d64cdbb28a3cf90a64676f122498ae99c76976f8067863ab00fcf3" Sep 06 05:58:28.083170 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:28.083129 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8x82j_openshift-console-operator(c23eec1b-1145-4e2d-9d9f-4b76157f40a0)\"" pod="openshift-console-operator/console-operator-85b454874b-8x82j" podUID="c23eec1b-1145-4e2d-9d9f-4b76157f40a0" Sep 06 05:58:28.218478 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:28.218280 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:28.218478 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:28.218334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:28.218478 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:28.218382 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 06 05:58:28.218478 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:28.218412 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 06 05:58:28.218478 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:28.218450 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm podName:c05aa74a-9686-4a1c-a26e-68ef0f755859 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:29.218430632 +0000 UTC m=+274.616761092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llpf2" (UID: "c05aa74a-9686-4a1c-a26e-68ef0f755859") : configmap "kube-rbac-proxy" not found Sep 06 05:58:28.218869 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:28.218512 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls podName:c05aa74a-9686-4a1c-a26e-68ef0f755859 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:29.218500468 +0000 UTC m=+274.616830914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llpf2" (UID: "c05aa74a-9686-4a1c-a26e-68ef0f755859") : secret "insights-runtime-extractor-tls" not found Sep 06 05:58:29.087523 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:29.087497 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/1.log" Sep 06 05:58:29.088009 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:29.087937 2578 scope.go:117] "RemoveContainer" containerID="37b8933f23d64cdbb28a3cf90a64676f122498ae99c76976f8067863ab00fcf3" Sep 06 05:58:29.088154 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:29.088133 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8x82j_openshift-console-operator(c23eec1b-1145-4e2d-9d9f-4b76157f40a0)\"" pod="openshift-console-operator/console-operator-85b454874b-8x82j" podUID="c23eec1b-1145-4e2d-9d9f-4b76157f40a0" Sep 06 05:58:29.225120 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:29.225069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:29.225337 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:29.225156 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 06 05:58:29.225337 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:29.225219 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:29.225337 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:29.225276 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 06 05:58:29.225337 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:29.225288 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm podName:c05aa74a-9686-4a1c-a26e-68ef0f755859 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:31.225268345 +0000 UTC m=+276.623598791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llpf2" (UID: "c05aa74a-9686-4a1c-a26e-68ef0f755859") : configmap "kube-rbac-proxy" not found Sep 06 05:58:29.225337 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:29.225318 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls podName:c05aa74a-9686-4a1c-a26e-68ef0f755859 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:31.225307299 +0000 UTC m=+276.623637745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llpf2" (UID: "c05aa74a-9686-4a1c-a26e-68ef0f755859") : secret "insights-runtime-extractor-tls" not found Sep 06 05:58:29.265639 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:29.265612 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qcvmz_e97d5427-3253-45fd-8a75-b0f4a8dd0452/dns-node-resolver/0.log" Sep 06 05:58:30.266213 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:30.266191 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6fw4z_bddf45b0-9199-4a9d-aee3-ca253d8a9c56/node-ca/0.log" Sep 06 05:58:30.434118 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:30.434088 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:30.434294 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:30.434167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:30.434294 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:30.434236 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 06 05:58:30.434403 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:30.434302 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs podName:c9644473-eedd-4380-a23d-ecf5b0c85f10 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:38.434283085 +0000 UTC m=+283.832613535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs") pod "router-default-99f956f9b-9d6ht" (UID: "c9644473-eedd-4380-a23d-ecf5b0c85f10") : secret "router-metrics-certs-default" not found Sep 06 05:58:30.434403 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:30.434316 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle podName:c9644473-eedd-4380-a23d-ecf5b0c85f10 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:38.434310562 +0000 UTC m=+283.832641008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle") pod "router-default-99f956f9b-9d6ht" (UID: "c9644473-eedd-4380-a23d-ecf5b0c85f10") : configmap references non-existent config key: service-ca.crt Sep 06 05:58:31.240641 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:31.240599 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:31.240641 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:31.240640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:31.240854 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:31.240723 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 06 05:58:31.240854 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:31.240785 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 06 05:58:31.240854 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:31.240791 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm podName:c05aa74a-9686-4a1c-a26e-68ef0f755859 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:35.240775294 +0000 UTC m=+280.639105741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llpf2" (UID: "c05aa74a-9686-4a1c-a26e-68ef0f755859") : configmap "kube-rbac-proxy" not found Sep 06 05:58:31.240854 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:31.240822 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls podName:c05aa74a-9686-4a1c-a26e-68ef0f755859 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:35.240811208 +0000 UTC m=+280.639141654 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llpf2" (UID: "c05aa74a-9686-4a1c-a26e-68ef0f755859") : secret "insights-runtime-extractor-tls" not found Sep 06 05:58:31.265118 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:31.265093 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-trktn_df6097ca-4ae8-47d6-918b-189f75f655cd/serve-healthcheck-canary/0.log" Sep 06 05:58:33.083715 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:33.083685 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:33.084193 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:33.083746 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:33.084193 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:33.084147 2578 scope.go:117] "RemoveContainer" containerID="37b8933f23d64cdbb28a3cf90a64676f122498ae99c76976f8067863ab00fcf3" Sep 06 05:58:33.084362 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:33.084339 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8x82j_openshift-console-operator(c23eec1b-1145-4e2d-9d9f-4b76157f40a0)\"" pod="openshift-console-operator/console-operator-85b454874b-8x82j" podUID="c23eec1b-1145-4e2d-9d9f-4b76157f40a0" Sep 06 05:58:35.273531 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:35.273495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:35.273531 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:35.273528 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:35.273980 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:35.273607 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 06 05:58:35.273980 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:35.273670 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm podName:c05aa74a-9686-4a1c-a26e-68ef0f755859 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:43.273655097 +0000 UTC m=+288.671985542 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llpf2" (UID: "c05aa74a-9686-4a1c-a26e-68ef0f755859") : configmap "kube-rbac-proxy" not found Sep 06 05:58:35.273980 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:35.273677 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 06 05:58:35.273980 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:35.273715 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls podName:c05aa74a-9686-4a1c-a26e-68ef0f755859 nodeName:}" failed. No retries permitted until 2026-09-06 05:58:43.273704297 +0000 UTC m=+288.672034742 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llpf2" (UID: "c05aa74a-9686-4a1c-a26e-68ef0f755859") : secret "insights-runtime-extractor-tls" not found Sep 06 05:58:35.773645 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:35.773611 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zmlqw" podUID="fee5736d-8984-42af-9121-8fe5d123a3ce" Sep 06 05:58:35.773796 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:35.773611 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6f46956478-tn984" podUID="5edfb5f6-d9a0-4e21-8195-6be1703189fb" Sep 06 05:58:36.113278 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:36.113204 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:58:36.113278 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:36.113244 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmlqw" Sep 06 05:58:38.507258 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:38.507220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:38.507612 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:38.507305 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:38.507815 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:38.507793 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9644473-eedd-4380-a23d-ecf5b0c85f10-service-ca-bundle\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:38.509698 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:38.509673 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9644473-eedd-4380-a23d-ecf5b0c85f10-metrics-certs\") pod \"router-default-99f956f9b-9d6ht\" (UID: \"c9644473-eedd-4380-a23d-ecf5b0c85f10\") " pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:38.580504 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:38.580472 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:38.851886 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:38.851862 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-99f956f9b-9d6ht"] Sep 06 05:58:38.854131 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:58:38.854108 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9644473_eedd_4380_a23d_ecf5b0c85f10.slice/crio-475a45a0e4d7f16507156b7723a2e4b80a9745b37a096d3ba369493f49565a46 WatchSource:0}: Error finding container 475a45a0e4d7f16507156b7723a2e4b80a9745b37a096d3ba369493f49565a46: Status 404 returned error can't find the container with id 475a45a0e4d7f16507156b7723a2e4b80a9745b37a096d3ba369493f49565a46 Sep 06 05:58:39.123805 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:39.123727 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-99f956f9b-9d6ht" event={"ID":"c9644473-eedd-4380-a23d-ecf5b0c85f10","Type":"ContainerStarted","Data":"100180348bc462bca3d1791845a5bf520fe7e9c2989eb72233074cbaad114c6a"} Sep 06 05:58:39.123805 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:39.123784 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-99f956f9b-9d6ht" event={"ID":"c9644473-eedd-4380-a23d-ecf5b0c85f10","Type":"ContainerStarted","Data":"475a45a0e4d7f16507156b7723a2e4b80a9745b37a096d3ba369493f49565a46"} Sep 06 05:58:39.141359 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:39.141312 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-99f956f9b-9d6ht" podStartSLOduration=17.141297766 podStartE2EDuration="17.141297766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:39.141031814 +0000 UTC m=+284.539362283" watchObservedRunningTime="2026-09-06 05:58:39.141297766 +0000 UTC m=+284.539628234" Sep 06 05:58:39.515085 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:39.514997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:58:39.515085 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:39.515039 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:58:39.517539 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:39.517513 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fee5736d-8984-42af-9121-8fe5d123a3ce-metrics-tls\") pod \"dns-default-zmlqw\" (UID: \"fee5736d-8984-42af-9121-8fe5d123a3ce\") " pod="openshift-dns/dns-default-zmlqw" Sep 06 05:58:39.517539 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:39.517525 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls\") pod \"image-registry-6f46956478-tn984\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:58:39.580962 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:39.580928 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:39.583487 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:39.583463 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:39.716483 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:39.716450 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qcp4t\"" Sep 06 05:58:39.716637 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:39.716508 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ppscw\"" Sep 06 05:58:39.725278 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:39.725257 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmlqw" Sep 06 05:58:39.725388 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:39.725355 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:58:40.001475 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:40.001444 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zmlqw"] Sep 06 05:58:40.004283 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:40.004259 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f46956478-tn984"] Sep 06 05:58:40.004667 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:58:40.004637 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee5736d_8984_42af_9121_8fe5d123a3ce.slice/crio-48dbf4f4199955888322f67df181fce74f24a5b5334056e5b911df6c802691f9 WatchSource:0}: Error finding container 48dbf4f4199955888322f67df181fce74f24a5b5334056e5b911df6c802691f9: Status 404 returned error can't find the container with id 48dbf4f4199955888322f67df181fce74f24a5b5334056e5b911df6c802691f9 Sep 06 05:58:40.007332 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:58:40.007312 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edfb5f6_d9a0_4e21_8195_6be1703189fb.slice/crio-2e9bfead594af4aac5f310a04cd1056cdf5854009fb7e9916f917be9af67757d WatchSource:0}: Error finding container 2e9bfead594af4aac5f310a04cd1056cdf5854009fb7e9916f917be9af67757d: Status 404 returned error can't find the container with id 2e9bfead594af4aac5f310a04cd1056cdf5854009fb7e9916f917be9af67757d Sep 06 05:58:40.128316 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:40.128286 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmlqw" event={"ID":"fee5736d-8984-42af-9121-8fe5d123a3ce","Type":"ContainerStarted","Data":"48dbf4f4199955888322f67df181fce74f24a5b5334056e5b911df6c802691f9"} Sep 06 05:58:40.129980 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:40.129953 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f46956478-tn984" event={"ID":"5edfb5f6-d9a0-4e21-8195-6be1703189fb","Type":"ContainerStarted","Data":"f24a201daef22c05e8f22edb84212fb9c1148d1623ccda846c3f63b605b99f1d"} Sep 06 05:58:40.130089 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:40.129994 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f46956478-tn984" event={"ID":"5edfb5f6-d9a0-4e21-8195-6be1703189fb","Type":"ContainerStarted","Data":"2e9bfead594af4aac5f310a04cd1056cdf5854009fb7e9916f917be9af67757d"} Sep 06 05:58:40.130303 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:40.130276 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:40.131491 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:40.131474 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-99f956f9b-9d6ht" Sep 06 05:58:40.149710 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:40.149671 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f46956478-tn984" podStartSLOduration=263.14965741 podStartE2EDuration="4m23.14965741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:40.148635045 +0000 UTC m=+285.546965576" watchObservedRunningTime="2026-09-06 05:58:40.14965741 +0000 UTC m=+285.547987878" Sep 06 05:58:41.133400 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:41.133360 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:58:42.138288 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:42.138252 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmlqw" event={"ID":"fee5736d-8984-42af-9121-8fe5d123a3ce","Type":"ContainerStarted","Data":"d9d6d735ee6f5adb9f47c8663a206046098d36915855fca685a28612d7af0c37"} Sep 06 05:58:42.138657 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:42.138298 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmlqw" event={"ID":"fee5736d-8984-42af-9121-8fe5d123a3ce","Type":"ContainerStarted","Data":"f60009a8e753c9269ea823ade455786d2feff11fc79faa39fac4afa85ec79a68"} Sep 06 05:58:42.138657 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:42.138539 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zmlqw" Sep 06 05:58:42.155649 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:42.155598 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zmlqw" podStartSLOduration=251.761321698 podStartE2EDuration="4m13.155581621s" totalImagesPullingTime="1.394259923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:42.154562122 +0000 UTC m=+287.552892602" watchObservedRunningTime="2026-09-06 05:58:42.155581621 +0000 UTC m=+287.553912107" Sep 06 05:58:43.348225 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:43.348182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:43.348225 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:43.348225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:43.348662 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:43.348642 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c05aa74a-9686-4a1c-a26e-68ef0f755859-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:43.350609 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:43.350593 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c05aa74a-9686-4a1c-a26e-68ef0f755859-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llpf2\" (UID: \"c05aa74a-9686-4a1c-a26e-68ef0f755859\") " pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:43.443398 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:43.443363 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-llpf2" Sep 06 05:58:43.711442 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:43.711294 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-llpf2"] Sep 06 05:58:43.714110 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:58:43.714084 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05aa74a_9686_4a1c_a26e_68ef0f755859.slice/crio-b92ad2cb6b797c6259c1a035185a89644190e5b99c554fc6e58d29c44ebdb9ad WatchSource:0}: Error finding container b92ad2cb6b797c6259c1a035185a89644190e5b99c554fc6e58d29c44ebdb9ad: Status 404 returned error can't find the container with id b92ad2cb6b797c6259c1a035185a89644190e5b99c554fc6e58d29c44ebdb9ad Sep 06 05:58:44.150346 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:44.150310 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llpf2" event={"ID":"c05aa74a-9686-4a1c-a26e-68ef0f755859","Type":"ContainerStarted","Data":"e0c79713be9802676e4d8097af46819899c507cb8ddcd332306ecc9cfa9556f6"} Sep 06 05:58:44.150512 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:44.150384 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llpf2" event={"ID":"c05aa74a-9686-4a1c-a26e-68ef0f755859","Type":"ContainerStarted","Data":"b92ad2cb6b797c6259c1a035185a89644190e5b99c554fc6e58d29c44ebdb9ad"} Sep 06 05:58:45.155242 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:45.155203 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llpf2" event={"ID":"c05aa74a-9686-4a1c-a26e-68ef0f755859","Type":"ContainerStarted","Data":"875fb23fc5c993b35c50ba0455ca2cc0a6d9610605cba55ce40634cfd0e6b44b"} Sep 06 05:58:46.139121 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:46.139094 2578 scope.go:117] "RemoveContainer" containerID="37b8933f23d64cdbb28a3cf90a64676f122498ae99c76976f8067863ab00fcf3" Sep 06 05:58:46.161237 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:46.161214 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llpf2" event={"ID":"c05aa74a-9686-4a1c-a26e-68ef0f755859","Type":"ContainerStarted","Data":"b29854a69ba3e4634680503f88e645a1c441e34049a5b99defec5c0395baecbc"} Sep 06 05:58:46.192968 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:46.192927 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-llpf2" podStartSLOduration=17.06325754 podStartE2EDuration="19.192889071s" totalImagesPullingTime="2.129631531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 05:58:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:46.192120592 +0000 UTC m=+291.590451076" watchObservedRunningTime="2026-09-06 05:58:46.192889071 +0000 UTC m=+291.591219538" Sep 06 05:58:47.166161 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:47.166133 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 05:58:47.166552 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:47.166496 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/1.log" Sep 06 05:58:47.166552 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:47.166525 2578 generic.go:356] "Generic (PLEG): container finished" podID="c23eec1b-1145-4e2d-9d9f-4b76157f40a0" containerID="b82c486ea33dd8fe01d4517b017d418163954a8af5b2f66cbc6791782fb9c207" exitCode=255 Sep 06 05:58:47.166627 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:47.166595 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8x82j" event={"ID":"c23eec1b-1145-4e2d-9d9f-4b76157f40a0","Type":"ContainerDied","Data":"b82c486ea33dd8fe01d4517b017d418163954a8af5b2f66cbc6791782fb9c207"} Sep 06 05:58:47.166676 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:47.166644 2578 scope.go:117] "RemoveContainer" containerID="37b8933f23d64cdbb28a3cf90a64676f122498ae99c76976f8067863ab00fcf3" Sep 06 05:58:47.167013 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:47.166990 2578 scope.go:117] "RemoveContainer" containerID="b82c486ea33dd8fe01d4517b017d418163954a8af5b2f66cbc6791782fb9c207" Sep 06 05:58:47.167226 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:47.167155 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-8x82j_openshift-console-operator(c23eec1b-1145-4e2d-9d9f-4b76157f40a0)\"" pod="openshift-console-operator/console-operator-85b454874b-8x82j" podUID="c23eec1b-1145-4e2d-9d9f-4b76157f40a0" Sep 06 05:58:47.365566 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:47.362804 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f46956478-tn984"] Sep 06 05:58:48.171527 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:48.171498 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 05:58:50.200044 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.200010 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-nxrrr"] Sep 06 05:58:50.204325 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.204310 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" Sep 06 05:58:50.206405 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.206378 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 06 05:58:50.206565 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.206540 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 06 05:58:50.206683 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.206542 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 06 05:58:50.207038 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.207023 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6kmtl\"" Sep 06 05:58:50.207141 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.207041 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 06 05:58:50.207209 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.207152 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 06 05:58:50.211302 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.211278 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-nxrrr"] Sep 06 05:58:50.298885 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.298855 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2727d715-b3ee-442d-90eb-88773241e40f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-nxrrr\" (UID: \"2727d715-b3ee-442d-90eb-88773241e40f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" Sep 06 05:58:50.298885 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.298884 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f2sh\" (UniqueName: \"kubernetes.io/projected/2727d715-b3ee-442d-90eb-88773241e40f-kube-api-access-2f2sh\") pod \"prometheus-operator-6f4946db56-nxrrr\" (UID: \"2727d715-b3ee-442d-90eb-88773241e40f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" Sep 06 05:58:50.299093 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.298989 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2727d715-b3ee-442d-90eb-88773241e40f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-nxrrr\" (UID: \"2727d715-b3ee-442d-90eb-88773241e40f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" Sep 06 05:58:50.299093 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.299040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2727d715-b3ee-442d-90eb-88773241e40f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-nxrrr\" (UID: \"2727d715-b3ee-442d-90eb-88773241e40f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" Sep 06 05:58:50.400283 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.400248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2727d715-b3ee-442d-90eb-88773241e40f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-nxrrr\" (UID: \"2727d715-b3ee-442d-90eb-88773241e40f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" Sep 06 05:58:50.400404 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.400285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2727d715-b3ee-442d-90eb-88773241e40f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-nxrrr\" (UID: \"2727d715-b3ee-442d-90eb-88773241e40f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" Sep 06 05:58:50.400404 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.400312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2727d715-b3ee-442d-90eb-88773241e40f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-nxrrr\" (UID: \"2727d715-b3ee-442d-90eb-88773241e40f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" Sep 06 05:58:50.400404 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.400329 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f2sh\" (UniqueName: \"kubernetes.io/projected/2727d715-b3ee-442d-90eb-88773241e40f-kube-api-access-2f2sh\") pod \"prometheus-operator-6f4946db56-nxrrr\" (UID: \"2727d715-b3ee-442d-90eb-88773241e40f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" Sep 06 05:58:50.400979 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.400958 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2727d715-b3ee-442d-90eb-88773241e40f-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-nxrrr\" (UID: \"2727d715-b3ee-442d-90eb-88773241e40f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" Sep 06 05:58:50.402999 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.402980 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2727d715-b3ee-442d-90eb-88773241e40f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-nxrrr\" (UID: \"2727d715-b3ee-442d-90eb-88773241e40f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" Sep 06 05:58:50.403092 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.403070 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2727d715-b3ee-442d-90eb-88773241e40f-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-nxrrr\" (UID: \"2727d715-b3ee-442d-90eb-88773241e40f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" Sep 06 05:58:50.407180 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.407155 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f2sh\" (UniqueName: \"kubernetes.io/projected/2727d715-b3ee-442d-90eb-88773241e40f-kube-api-access-2f2sh\") pod \"prometheus-operator-6f4946db56-nxrrr\" (UID: \"2727d715-b3ee-442d-90eb-88773241e40f\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" Sep 06 05:58:50.514747 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.514665 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" Sep 06 05:58:50.781390 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:50.781359 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-nxrrr"] Sep 06 05:58:50.784321 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:58:50.784294 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2727d715_b3ee_442d_90eb_88773241e40f.slice/crio-b3fc79dfb88815c0b574ede6441ad0586bcd75a43a6b93289d2e702808c52ac0 WatchSource:0}: Error finding container b3fc79dfb88815c0b574ede6441ad0586bcd75a43a6b93289d2e702808c52ac0: Status 404 returned error can't find the container with id b3fc79dfb88815c0b574ede6441ad0586bcd75a43a6b93289d2e702808c52ac0 Sep 06 05:58:51.181362 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:51.181279 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" event={"ID":"2727d715-b3ee-442d-90eb-88773241e40f","Type":"ContainerStarted","Data":"b3fc79dfb88815c0b574ede6441ad0586bcd75a43a6b93289d2e702808c52ac0"} Sep 06 05:58:52.139084 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:52.139043 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-w74pq" podUID="f36acd55-4ce4-4bdb-9518-cca261651a19" Sep 06 05:58:52.144712 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:52.144688 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zmlqw" Sep 06 05:58:52.191856 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:52.191241 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" event={"ID":"2727d715-b3ee-442d-90eb-88773241e40f","Type":"ContainerStarted","Data":"d59fd94316184b10fa0aa78661b5023aa937c2c56c4a0eaef0542be404923e71"} Sep 06 05:58:52.191856 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:52.191305 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" event={"ID":"2727d715-b3ee-442d-90eb-88773241e40f","Type":"ContainerStarted","Data":"83331abfa6c7ae1bf33f2bea0ee8dfcdb0d44b612d2d20c08b635720ff06dab8"} Sep 06 05:58:52.206351 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:52.206254 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-nxrrr" podStartSLOduration=1.043463529 podStartE2EDuration="2.206237061s" totalImagesPullingTime="1.162773532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:52.205704361 +0000 UTC m=+297.604034891" watchObservedRunningTime="2026-09-06 05:58:52.206237061 +0000 UTC m=+297.604567547" Sep 06 05:58:53.084049 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:53.084008 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:53.084272 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:53.084096 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:58:53.084414 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:53.084401 2578 scope.go:117] "RemoveContainer" containerID="b82c486ea33dd8fe01d4517b017d418163954a8af5b2f66cbc6791782fb9c207" Sep 06 05:58:53.084591 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:53.084572 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-8x82j_openshift-console-operator(c23eec1b-1145-4e2d-9d9f-4b76157f40a0)\"" pod="openshift-console-operator/console-operator-85b454874b-8x82j" podUID="c23eec1b-1145-4e2d-9d9f-4b76157f40a0" Sep 06 05:58:54.576726 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.576693 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rcmg5"] Sep 06 05:58:54.582785 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.582770 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.584464 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.584434 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pdq8f\"" Sep 06 05:58:54.584582 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.584529 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 06 05:58:54.584582 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.584540 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 06 05:58:54.584797 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.584782 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 06 05:58:54.739030 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.738996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-node-exporter-textfile\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.739193 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.739040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-node-exporter-tls\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.739193 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.739063 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.739193 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.739082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-sys\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.739193 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.739146 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-node-exporter-wtmp\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.739193 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.739184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-metrics-client-ca\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.739353 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.739234 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.739353 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.739257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qlpm\" (UniqueName: \"kubernetes.io/projected/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-kube-api-access-8qlpm\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.739353 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.739289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-root\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.840675 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.840595 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-root\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.840675 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.840628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-node-exporter-textfile\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.840675 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.840660 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-node-exporter-tls\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.840954 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.840687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.840954 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.840716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-sys\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.840954 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.840748 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-sys\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.840954 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.840749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-node-exporter-wtmp\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.840954 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.840715 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-root\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.840954 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.840801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-metrics-client-ca\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.840954 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.840859 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-node-exporter-wtmp\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.841225 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.840952 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.841225 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.840995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qlpm\" (UniqueName: \"kubernetes.io/projected/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-kube-api-access-8qlpm\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.841225 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.841051 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-node-exporter-textfile\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.841402 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.841380 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-metrics-client-ca\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.841461 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.841380 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-node-exporter-accelerators-collector-config\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.843139 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.843119 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.843239 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.843223 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-node-exporter-tls\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.849001 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.848981 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qlpm\" (UniqueName: \"kubernetes.io/projected/4a1ec7c7-c34f-4047-89fd-9cc56b7262a3-kube-api-access-8qlpm\") pod \"node-exporter-rcmg5\" (UID: \"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3\") " pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.891915 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:54.891876 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rcmg5" Sep 06 05:58:54.903796 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:58:54.903775 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1ec7c7_c34f_4047_89fd_9cc56b7262a3.slice/crio-614848b4412ef0451b3f9553a76c8a5e8b5f78bd6371afc069594ed0a52110b8 WatchSource:0}: Error finding container 614848b4412ef0451b3f9553a76c8a5e8b5f78bd6371afc069594ed0a52110b8: Status 404 returned error can't find the container with id 614848b4412ef0451b3f9553a76c8a5e8b5f78bd6371afc069594ed0a52110b8 Sep 06 05:58:55.030355 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.030323 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 05:58:55.031102 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.031081 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 05:58:55.040483 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.040465 2578 kubelet.go:1648] "Image garbage collection succeeded" Sep 06 05:58:55.201734 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.201551 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rcmg5" event={"ID":"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3","Type":"ContainerStarted","Data":"614848b4412ef0451b3f9553a76c8a5e8b5f78bd6371afc069594ed0a52110b8"} Sep 06 05:58:55.612747 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.612714 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 05:58:55.617157 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.617125 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.619775 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.619390 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 06 05:58:55.619775 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.619429 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 06 05:58:55.619775 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.619391 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 06 05:58:55.619775 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.619554 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 06 05:58:55.619775 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.619556 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 06 05:58:55.619775 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.619735 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 06 05:58:55.620209 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.619851 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 06 05:58:55.620209 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.619924 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zkwr2\"" Sep 06 05:58:55.620209 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.619993 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 06 05:58:55.620209 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.620081 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 06 05:58:55.629602 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.629578 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 05:58:55.748986 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.748958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.748986 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.748991 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-config-volume\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.749181 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.749033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b8705aaf-1985-4467-913b-92faeb227d4b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.749181 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.749049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trtbv\" (UniqueName: \"kubernetes.io/projected/b8705aaf-1985-4467-913b-92faeb227d4b-kube-api-access-trtbv\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.749181 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.749094 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.749181 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.749141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.749181 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.749163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8705aaf-1985-4467-913b-92faeb227d4b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.749181 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.749180 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.749358 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.749204 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8705aaf-1985-4467-913b-92faeb227d4b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.749358 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.749232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.749358 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.749261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-web-config\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.749358 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.749277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8705aaf-1985-4467-913b-92faeb227d4b-config-out\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.749358 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.749292 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8705aaf-1985-4467-913b-92faeb227d4b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.850483 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.850447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.850646 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.850493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.850646 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:55.850608 2578 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 06 05:58:55.850646 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.850622 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8705aaf-1985-4467-913b-92faeb227d4b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.850823 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.850662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.850823 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:58:55.850681 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-main-tls podName:b8705aaf-1985-4467-913b-92faeb227d4b nodeName:}" failed. No retries permitted until 2026-09-06 05:58:56.350657419 +0000 UTC m=+301.748987870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "b8705aaf-1985-4467-913b-92faeb227d4b") : secret "alertmanager-main-tls" not found Sep 06 05:58:55.850823 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.850722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8705aaf-1985-4467-913b-92faeb227d4b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.850823 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.850757 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.850823 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.850799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-web-config\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.851124 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.850825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8705aaf-1985-4467-913b-92faeb227d4b-config-out\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.851124 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.850848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8705aaf-1985-4467-913b-92faeb227d4b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.851124 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.850919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.851124 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.850952 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-config-volume\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.851124 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.851016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b8705aaf-1985-4467-913b-92faeb227d4b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.851124 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.851037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trtbv\" (UniqueName: \"kubernetes.io/projected/b8705aaf-1985-4467-913b-92faeb227d4b-kube-api-access-trtbv\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.851586 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.851399 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8705aaf-1985-4467-913b-92faeb227d4b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.851961 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.851914 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b8705aaf-1985-4467-913b-92faeb227d4b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.852665 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.852388 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8705aaf-1985-4467-913b-92faeb227d4b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.854875 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.854848 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-web-config\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.855065 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.855036 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.855211 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.855183 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.855421 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.855396 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.855543 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.855513 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-config-volume\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.855543 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.855513 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8705aaf-1985-4467-913b-92faeb227d4b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.855978 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.855960 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.856193 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.856171 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8705aaf-1985-4467-913b-92faeb227d4b-config-out\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:55.860334 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:55.860311 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trtbv\" (UniqueName: \"kubernetes.io/projected/b8705aaf-1985-4467-913b-92faeb227d4b-kube-api-access-trtbv\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:56.206761 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:56.206729 2578 generic.go:356] "Generic (PLEG): container finished" podID="4a1ec7c7-c34f-4047-89fd-9cc56b7262a3" containerID="547aed965e1d0f498782a4bb2342569b76f315190b9aa8aed380c4f3f4072bec" exitCode=0 Sep 06 05:58:56.206956 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:56.206778 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rcmg5" event={"ID":"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3","Type":"ContainerDied","Data":"547aed965e1d0f498782a4bb2342569b76f315190b9aa8aed380c4f3f4072bec"} Sep 06 05:58:56.354512 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:56.354486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:56.356984 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:56.356962 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:56.528167 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:56.528135 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 06 05:58:56.809466 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:56.809442 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 05:58:56.810997 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:58:56.810973 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8705aaf_1985_4467_913b_92faeb227d4b.slice/crio-44e9419e2b4ee876b0d0c4ad4f238920d167b44765a1829bf8d989133682c51b WatchSource:0}: Error finding container 44e9419e2b4ee876b0d0c4ad4f238920d167b44765a1829bf8d989133682c51b: Status 404 returned error can't find the container with id 44e9419e2b4ee876b0d0c4ad4f238920d167b44765a1829bf8d989133682c51b Sep 06 05:58:56.813086 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:56.813068 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 05:58:57.211675 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:57.211585 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8705aaf-1985-4467-913b-92faeb227d4b","Type":"ContainerStarted","Data":"44e9419e2b4ee876b0d0c4ad4f238920d167b44765a1829bf8d989133682c51b"} Sep 06 05:58:57.213717 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:57.213694 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rcmg5" event={"ID":"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3","Type":"ContainerStarted","Data":"4ef6a546d7538706b0672a2eedaec453eaa096d3499c24837f0cac0bdc18de25"} Sep 06 05:58:57.213830 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:57.213732 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rcmg5" event={"ID":"4a1ec7c7-c34f-4047-89fd-9cc56b7262a3","Type":"ContainerStarted","Data":"75f4908734945c87443e76a31ccac6b821d4fd3aefbe77a1458b2cd4d1951798"} Sep 06 05:58:57.235334 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:57.235289 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rcmg5" podStartSLOduration=2.043813272 podStartE2EDuration="3.23527425s" totalImagesPullingTime="1.123127642s" totalInitContainerRuntime="68.333336ms" isStatefulPod=true podCreationTimestamp="2026-09-06 05:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:58:57.234101401 +0000 UTC m=+302.632431869" watchObservedRunningTime="2026-09-06 05:58:57.23527425 +0000 UTC m=+302.633604718" Sep 06 05:58:57.370085 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:57.370021 2578 patch_prober.go:28] interesting pod/image-registry-6f46956478-tn984 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 06 05:58:57.370339 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:57.370100 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6f46956478-tn984" podUID="5edfb5f6-d9a0-4e21-8195-6be1703189fb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 06 05:58:58.219630 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:58.219535 2578 generic.go:356] "Generic (PLEG): container finished" podID="b8705aaf-1985-4467-913b-92faeb227d4b" containerID="6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1" exitCode=0 Sep 06 05:58:58.220039 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:58.219626 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8705aaf-1985-4467-913b-92faeb227d4b","Type":"ContainerDied","Data":"6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1"} Sep 06 05:58:59.331586 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:59.331555 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw"] Sep 06 05:58:59.336427 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:59.336402 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw" Sep 06 05:58:59.338382 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:59.338358 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 06 05:58:59.338506 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:59.338403 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-g69r8\"" Sep 06 05:58:59.344856 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:59.344831 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw"] Sep 06 05:58:59.377325 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:59.377296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2630fd07-dd9f-426a-9d5a-b49afec19668-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-f7ngw\" (UID: \"2630fd07-dd9f-426a-9d5a-b49afec19668\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw" Sep 06 05:58:59.377510 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:59.377353 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtnz2\" (UniqueName: \"kubernetes.io/projected/2630fd07-dd9f-426a-9d5a-b49afec19668-kube-api-access-jtnz2\") pod \"monitoring-plugin-85bbbbbf4-f7ngw\" (UID: \"2630fd07-dd9f-426a-9d5a-b49afec19668\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw" Sep 06 05:58:59.478740 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:59.478703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnz2\" (UniqueName: \"kubernetes.io/projected/2630fd07-dd9f-426a-9d5a-b49afec19668-kube-api-access-jtnz2\") pod \"monitoring-plugin-85bbbbbf4-f7ngw\" (UID: \"2630fd07-dd9f-426a-9d5a-b49afec19668\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw" Sep 06 05:58:59.478925 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:59.478840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2630fd07-dd9f-426a-9d5a-b49afec19668-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-f7ngw\" (UID: \"2630fd07-dd9f-426a-9d5a-b49afec19668\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw" Sep 06 05:58:59.481752 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:59.481722 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2630fd07-dd9f-426a-9d5a-b49afec19668-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-f7ngw\" (UID: \"2630fd07-dd9f-426a-9d5a-b49afec19668\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw" Sep 06 05:58:59.486718 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:59.486694 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtnz2\" (UniqueName: \"kubernetes.io/projected/2630fd07-dd9f-426a-9d5a-b49afec19668-kube-api-access-jtnz2\") pod \"monitoring-plugin-85bbbbbf4-f7ngw\" (UID: \"2630fd07-dd9f-426a-9d5a-b49afec19668\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw" Sep 06 05:58:59.650395 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:58:59.650304 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw" Sep 06 05:59:00.072462 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:00.072439 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw"] Sep 06 05:59:00.169162 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:59:00.169125 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2630fd07_dd9f_426a_9d5a_b49afec19668.slice/crio-e54268aab7d7e1eb06cb3341673ef26961049b4544e54926b33c0627e96cc2cb WatchSource:0}: Error finding container e54268aab7d7e1eb06cb3341673ef26961049b4544e54926b33c0627e96cc2cb: Status 404 returned error can't find the container with id e54268aab7d7e1eb06cb3341673ef26961049b4544e54926b33c0627e96cc2cb Sep 06 05:59:00.230005 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:00.229977 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw" event={"ID":"2630fd07-dd9f-426a-9d5a-b49afec19668","Type":"ContainerStarted","Data":"e54268aab7d7e1eb06cb3341673ef26961049b4544e54926b33c0627e96cc2cb"} Sep 06 05:59:01.237998 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:01.237920 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8705aaf-1985-4467-913b-92faeb227d4b","Type":"ContainerStarted","Data":"fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867"} Sep 06 05:59:01.237998 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:01.237984 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8705aaf-1985-4467-913b-92faeb227d4b","Type":"ContainerStarted","Data":"1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571"} Sep 06 05:59:01.238385 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:01.238010 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8705aaf-1985-4467-913b-92faeb227d4b","Type":"ContainerStarted","Data":"2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1"} Sep 06 05:59:01.238385 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:01.238034 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8705aaf-1985-4467-913b-92faeb227d4b","Type":"ContainerStarted","Data":"d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82"} Sep 06 05:59:01.238385 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:01.238057 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8705aaf-1985-4467-913b-92faeb227d4b","Type":"ContainerStarted","Data":"a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409"} Sep 06 05:59:03.138982 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:03.138939 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 05:59:03.246073 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:03.246030 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw" event={"ID":"2630fd07-dd9f-426a-9d5a-b49afec19668","Type":"ContainerStarted","Data":"4c3c2395e3fa56d19218958e9c6eaaef20acdded0226c11b92226370f4d709c9"} Sep 06 05:59:03.246265 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:03.246207 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw" Sep 06 05:59:03.249615 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:03.249591 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8705aaf-1985-4467-913b-92faeb227d4b","Type":"ContainerStarted","Data":"12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3"} Sep 06 05:59:03.251392 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:03.251374 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw" Sep 06 05:59:03.260506 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:03.260466 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-f7ngw" podStartSLOduration=2.156924416 podStartE2EDuration="4.260452584s" totalImagesPullingTime="2.103528168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:59:03.259582106 +0000 UTC m=+308.657912585" watchObservedRunningTime="2026-09-06 05:59:03.260452584 +0000 UTC m=+308.658783050" Sep 06 05:59:03.281419 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:03.281378 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.76500551 podStartE2EDuration="8.281361755s" totalImagesPullingTime="4.499892764s" totalInitContainerRuntime="16.463481ms" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:59:03.280517537 +0000 UTC m=+308.678848004" watchObservedRunningTime="2026-09-06 05:59:03.281361755 +0000 UTC m=+308.679692200" Sep 06 05:59:05.140617 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:05.140587 2578 scope.go:117] "RemoveContainer" containerID="b82c486ea33dd8fe01d4517b017d418163954a8af5b2f66cbc6791782fb9c207" Sep 06 05:59:05.141015 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:59:05.140762 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-8x82j_openshift-console-operator(c23eec1b-1145-4e2d-9d9f-4b76157f40a0)\"" pod="openshift-console-operator/console-operator-85b454874b-8x82j" podUID="c23eec1b-1145-4e2d-9d9f-4b76157f40a0" Sep 06 05:59:07.368075 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:07.368040 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:59:12.384175 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.384132 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f46956478-tn984" podUID="5edfb5f6-d9a0-4e21-8195-6be1703189fb" containerName="registry" containerID="cri-o://f24a201daef22c05e8f22edb84212fb9c1148d1623ccda846c3f63b605b99f1d" gracePeriod=30 Sep 06 05:59:12.622442 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.622420 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:59:12.697448 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.697371 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5edfb5f6-d9a0-4e21-8195-6be1703189fb-ca-trust-extracted\") pod \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " Sep 06 05:59:12.697448 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.697402 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-certificates\") pod \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " Sep 06 05:59:12.697448 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.697434 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5edfb5f6-d9a0-4e21-8195-6be1703189fb-image-registry-private-configuration\") pod \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " Sep 06 05:59:12.697700 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.697453 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-bound-sa-token\") pod \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " Sep 06 05:59:12.697700 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.697473 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-487n2\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-kube-api-access-487n2\") pod \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " Sep 06 05:59:12.697700 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.697601 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5edfb5f6-d9a0-4e21-8195-6be1703189fb-installation-pull-secrets\") pod \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " Sep 06 05:59:12.697700 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.697636 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5edfb5f6-d9a0-4e21-8195-6be1703189fb-trusted-ca\") pod \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " Sep 06 05:59:12.697700 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.697670 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls\") pod \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\" (UID: \"5edfb5f6-d9a0-4e21-8195-6be1703189fb\") " Sep 06 05:59:12.697971 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.697849 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5edfb5f6-d9a0-4e21-8195-6be1703189fb" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 05:59:12.698178 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.698130 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edfb5f6-d9a0-4e21-8195-6be1703189fb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5edfb5f6-d9a0-4e21-8195-6be1703189fb" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 05:59:12.700074 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.700016 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5edfb5f6-d9a0-4e21-8195-6be1703189fb" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 05:59:12.700177 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.700073 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5edfb5f6-d9a0-4e21-8195-6be1703189fb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5edfb5f6-d9a0-4e21-8195-6be1703189fb" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 05:59:12.700319 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.700293 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5edfb5f6-d9a0-4e21-8195-6be1703189fb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5edfb5f6-d9a0-4e21-8195-6be1703189fb" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 05:59:12.700368 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.700348 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5edfb5f6-d9a0-4e21-8195-6be1703189fb" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 05:59:12.700695 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.700680 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-kube-api-access-487n2" (OuterVolumeSpecName: "kube-api-access-487n2") pod "5edfb5f6-d9a0-4e21-8195-6be1703189fb" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb"). InnerVolumeSpecName "kube-api-access-487n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 05:59:12.706366 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.706339 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edfb5f6-d9a0-4e21-8195-6be1703189fb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5edfb5f6-d9a0-4e21-8195-6be1703189fb" (UID: "5edfb5f6-d9a0-4e21-8195-6be1703189fb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 05:59:12.798267 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.798229 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-bound-sa-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 05:59:12.798267 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.798259 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-487n2\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-kube-api-access-487n2\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 05:59:12.798267 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.798270 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5edfb5f6-d9a0-4e21-8195-6be1703189fb-installation-pull-secrets\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 05:59:12.798479 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.798280 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5edfb5f6-d9a0-4e21-8195-6be1703189fb-trusted-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 05:59:12.798479 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.798289 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-tls\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 05:59:12.798479 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.798296 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5edfb5f6-d9a0-4e21-8195-6be1703189fb-ca-trust-extracted\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 05:59:12.798479 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.798304 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5edfb5f6-d9a0-4e21-8195-6be1703189fb-registry-certificates\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 05:59:12.798479 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:12.798313 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5edfb5f6-d9a0-4e21-8195-6be1703189fb-image-registry-private-configuration\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 05:59:13.287269 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:13.287236 2578 generic.go:356] "Generic (PLEG): container finished" podID="5edfb5f6-d9a0-4e21-8195-6be1703189fb" containerID="f24a201daef22c05e8f22edb84212fb9c1148d1623ccda846c3f63b605b99f1d" exitCode=0 Sep 06 05:59:13.287440 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:13.287281 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f46956478-tn984" Sep 06 05:59:13.287440 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:13.287292 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f46956478-tn984" event={"ID":"5edfb5f6-d9a0-4e21-8195-6be1703189fb","Type":"ContainerDied","Data":"f24a201daef22c05e8f22edb84212fb9c1148d1623ccda846c3f63b605b99f1d"} Sep 06 05:59:13.287440 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:13.287333 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f46956478-tn984" event={"ID":"5edfb5f6-d9a0-4e21-8195-6be1703189fb","Type":"ContainerDied","Data":"2e9bfead594af4aac5f310a04cd1056cdf5854009fb7e9916f917be9af67757d"} Sep 06 05:59:13.287440 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:13.287352 2578 scope.go:117] "RemoveContainer" containerID="f24a201daef22c05e8f22edb84212fb9c1148d1623ccda846c3f63b605b99f1d" Sep 06 05:59:13.295418 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:13.295400 2578 scope.go:117] "RemoveContainer" containerID="f24a201daef22c05e8f22edb84212fb9c1148d1623ccda846c3f63b605b99f1d" Sep 06 05:59:13.295735 ip-10-0-135-32 kubenswrapper[2578]: E0906 05:59:13.295715 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f24a201daef22c05e8f22edb84212fb9c1148d1623ccda846c3f63b605b99f1d\": container with ID starting with f24a201daef22c05e8f22edb84212fb9c1148d1623ccda846c3f63b605b99f1d not found: ID does not exist" containerID="f24a201daef22c05e8f22edb84212fb9c1148d1623ccda846c3f63b605b99f1d" Sep 06 05:59:13.295788 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:13.295743 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f24a201daef22c05e8f22edb84212fb9c1148d1623ccda846c3f63b605b99f1d"} err="failed to get container status \"f24a201daef22c05e8f22edb84212fb9c1148d1623ccda846c3f63b605b99f1d\": rpc error: code = NotFound desc = could not find container \"f24a201daef22c05e8f22edb84212fb9c1148d1623ccda846c3f63b605b99f1d\": container with ID starting with f24a201daef22c05e8f22edb84212fb9c1148d1623ccda846c3f63b605b99f1d not found: ID does not exist" Sep 06 05:59:13.302743 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:13.302718 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f46956478-tn984"] Sep 06 05:59:13.307067 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:13.307046 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f46956478-tn984"] Sep 06 05:59:15.143357 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:15.143314 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5edfb5f6-d9a0-4e21-8195-6be1703189fb" path="/var/lib/kubelet/pods/5edfb5f6-d9a0-4e21-8195-6be1703189fb/volumes" Sep 06 05:59:17.139269 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.139241 2578 scope.go:117] "RemoveContainer" containerID="b82c486ea33dd8fe01d4517b017d418163954a8af5b2f66cbc6791782fb9c207" Sep 06 05:59:17.304673 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.304641 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 05:59:17.304852 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.304718 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8x82j" event={"ID":"c23eec1b-1145-4e2d-9d9f-4b76157f40a0","Type":"ContainerStarted","Data":"afb08c5eec29322b2e65129fc0b966312ec8d58d0b9548b3e29f28962b4842af"} Sep 06 05:59:17.305088 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.305071 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:59:17.319861 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.319808 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-8x82j" podStartSLOduration=52.159862971 podStartE2EDuration="55.319794765s" totalImagesPullingTime="3.159931794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:59:17.319640385 +0000 UTC m=+322.717970844" watchObservedRunningTime="2026-09-06 05:59:17.319794765 +0000 UTC m=+322.718125232" Sep 06 05:59:17.667230 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.667202 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-8x82j" Sep 06 05:59:17.844059 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.844012 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-hfnrt"] Sep 06 05:59:17.844522 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.844501 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5edfb5f6-d9a0-4e21-8195-6be1703189fb" containerName="registry" Sep 06 05:59:17.844573 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.844530 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5edfb5f6-d9a0-4e21-8195-6be1703189fb" containerName="registry" Sep 06 05:59:17.844693 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.844678 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5edfb5f6-d9a0-4e21-8195-6be1703189fb" containerName="registry" Sep 06 05:59:17.848072 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.848047 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-hfnrt" Sep 06 05:59:17.850235 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.850190 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 06 05:59:17.850394 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.850282 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 06 05:59:17.850394 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.850326 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-l265t\"" Sep 06 05:59:17.855111 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.855084 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-hfnrt"] Sep 06 05:59:17.940391 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:17.940312 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpwr\" (UniqueName: \"kubernetes.io/projected/87f80575-d356-4946-9532-39ba70525e7a-kube-api-access-hwpwr\") pod \"downloads-85d98c998f-hfnrt\" (UID: \"87f80575-d356-4946-9532-39ba70525e7a\") " pod="openshift-console/downloads-85d98c998f-hfnrt" Sep 06 05:59:18.041806 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:18.041772 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpwr\" (UniqueName: \"kubernetes.io/projected/87f80575-d356-4946-9532-39ba70525e7a-kube-api-access-hwpwr\") pod \"downloads-85d98c998f-hfnrt\" (UID: \"87f80575-d356-4946-9532-39ba70525e7a\") " pod="openshift-console/downloads-85d98c998f-hfnrt" Sep 06 05:59:18.052804 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:18.052772 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpwr\" (UniqueName: \"kubernetes.io/projected/87f80575-d356-4946-9532-39ba70525e7a-kube-api-access-hwpwr\") pod \"downloads-85d98c998f-hfnrt\" (UID: \"87f80575-d356-4946-9532-39ba70525e7a\") " pod="openshift-console/downloads-85d98c998f-hfnrt" Sep 06 05:59:18.159561 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:18.159532 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-hfnrt" Sep 06 05:59:18.431178 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:18.431156 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-hfnrt"] Sep 06 05:59:18.433651 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:59:18.433627 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f80575_d356_4946_9532_39ba70525e7a.slice/crio-52af48501d24d7f1ce6b255f12933f924fdd1cc58358fe76eae680e5d5429320 WatchSource:0}: Error finding container 52af48501d24d7f1ce6b255f12933f924fdd1cc58358fe76eae680e5d5429320: Status 404 returned error can't find the container with id 52af48501d24d7f1ce6b255f12933f924fdd1cc58358fe76eae680e5d5429320 Sep 06 05:59:19.314496 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:19.314462 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-hfnrt" event={"ID":"87f80575-d356-4946-9532-39ba70525e7a","Type":"ContainerStarted","Data":"52af48501d24d7f1ce6b255f12933f924fdd1cc58358fe76eae680e5d5429320"} Sep 06 05:59:35.383692 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:35.383650 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-hfnrt" event={"ID":"87f80575-d356-4946-9532-39ba70525e7a","Type":"ContainerStarted","Data":"77ed12a6d88ebe0f9f2023c3631d734afeea52c49d4c3891db56752e7e916a18"} Sep 06 05:59:35.384234 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:35.383894 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-hfnrt" Sep 06 05:59:35.399928 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:35.399852 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-hfnrt" podStartSLOduration=2.419253842 podStartE2EDuration="18.399832558s" totalImagesPullingTime="15.980578716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:59:35.398385789 +0000 UTC m=+340.796716268" watchObservedRunningTime="2026-09-06 05:59:35.399832558 +0000 UTC m=+340.798163023" Sep 06 05:59:35.407691 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:35.407661 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-hfnrt" Sep 06 05:59:36.950787 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:36.950746 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-789c8d65c9-lf2f7"] Sep 06 05:59:36.985710 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:36.985676 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-789c8d65c9-lf2f7"] Sep 06 05:59:36.985878 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:36.985813 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:36.988462 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:36.988437 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 06 05:59:36.988650 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:36.988619 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 06 05:59:36.989257 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:36.989237 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 06 05:59:36.989415 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:36.989284 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-whcwh\"" Sep 06 05:59:36.989646 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:36.989628 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 06 05:59:36.989764 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:36.989707 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 06 05:59:36.993922 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:36.993886 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 06 05:59:37.112002 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.111959 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e76934c1-092f-4826-900f-8a552f6eef21-console-oauth-config\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.112176 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.112035 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27lvx\" (UniqueName: \"kubernetes.io/projected/e76934c1-092f-4826-900f-8a552f6eef21-kube-api-access-27lvx\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.112176 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.112089 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e76934c1-092f-4826-900f-8a552f6eef21-console-serving-cert\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.112176 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.112106 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-console-config\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.112315 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.112177 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-oauth-serving-cert\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.112315 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.112208 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-trusted-ca-bundle\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.112315 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.112235 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-service-ca\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.213715 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.213631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-service-ca\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.213715 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.213679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e76934c1-092f-4826-900f-8a552f6eef21-console-oauth-config\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.213977 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.213733 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27lvx\" (UniqueName: \"kubernetes.io/projected/e76934c1-092f-4826-900f-8a552f6eef21-kube-api-access-27lvx\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.213977 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.213785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e76934c1-092f-4826-900f-8a552f6eef21-console-serving-cert\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.213977 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.213809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-console-config\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.213977 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.213847 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-oauth-serving-cert\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.213977 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.213918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-trusted-ca-bundle\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.214541 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.214512 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-service-ca\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.214755 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.214732 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-trusted-ca-bundle\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.214852 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.214740 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-console-config\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.214852 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.214791 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-oauth-serving-cert\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.216674 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.216653 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e76934c1-092f-4826-900f-8a552f6eef21-console-oauth-config\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.216831 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.216811 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e76934c1-092f-4826-900f-8a552f6eef21-console-serving-cert\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.223490 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.223464 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27lvx\" (UniqueName: \"kubernetes.io/projected/e76934c1-092f-4826-900f-8a552f6eef21-kube-api-access-27lvx\") pod \"console-789c8d65c9-lf2f7\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.298654 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.298618 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:37.619678 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:37.619498 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-789c8d65c9-lf2f7"] Sep 06 05:59:37.622647 ip-10-0-135-32 kubenswrapper[2578]: W0906 05:59:37.622615 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76934c1_092f_4826_900f_8a552f6eef21.slice/crio-62fc6f7fe24259fb3cdc99c47bf0c7f4725dfde925b444275c5607262583cab1 WatchSource:0}: Error finding container 62fc6f7fe24259fb3cdc99c47bf0c7f4725dfde925b444275c5607262583cab1: Status 404 returned error can't find the container with id 62fc6f7fe24259fb3cdc99c47bf0c7f4725dfde925b444275c5607262583cab1 Sep 06 05:59:38.399700 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:38.399624 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-789c8d65c9-lf2f7" event={"ID":"e76934c1-092f-4826-900f-8a552f6eef21","Type":"ContainerStarted","Data":"62fc6f7fe24259fb3cdc99c47bf0c7f4725dfde925b444275c5607262583cab1"} Sep 06 05:59:41.414538 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:41.414509 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-789c8d65c9-lf2f7" event={"ID":"e76934c1-092f-4826-900f-8a552f6eef21","Type":"ContainerStarted","Data":"6d01ca78f2559a8498541002e34e78becc35bc61044404231a9c716bb483f060"} Sep 06 05:59:42.421016 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:42.420972 2578 generic.go:356] "Generic (PLEG): container finished" podID="50fa393c-a0bf-4082-8040-8018f2ba260c" containerID="e234271a8c3574385afb3a6aed5e60f0d6e0e64ef480f5baf1b3ec8496268266" exitCode=0 Sep 06 05:59:42.421540 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:42.421051 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" event={"ID":"50fa393c-a0bf-4082-8040-8018f2ba260c","Type":"ContainerDied","Data":"e234271a8c3574385afb3a6aed5e60f0d6e0e64ef480f5baf1b3ec8496268266"} Sep 06 05:59:42.421540 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:42.421458 2578 scope.go:117] "RemoveContainer" containerID="e234271a8c3574385afb3a6aed5e60f0d6e0e64ef480f5baf1b3ec8496268266" Sep 06 05:59:42.438353 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:42.436224 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-789c8d65c9-lf2f7" podStartSLOduration=2.768471467 podStartE2EDuration="6.436206221s" totalImagesPullingTime="3.667734754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 05:59:42.435785014 +0000 UTC m=+347.834115485" watchObservedRunningTime="2026-09-06 05:59:42.436206221 +0000 UTC m=+347.834536690" Sep 06 05:59:43.427994 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:43.427951 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-6jcbw" event={"ID":"50fa393c-a0bf-4082-8040-8018f2ba260c","Type":"ContainerStarted","Data":"2a421481a09caa35b53c6aedd391780c5f56e48053a675574ba1ec4a95674307"} Sep 06 05:59:47.299097 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:47.299007 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:47.299097 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:47.299072 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:47.303756 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:47.303732 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:47.445994 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:47.445961 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 05:59:52.462030 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:52.461955 2578 generic.go:356] "Generic (PLEG): container finished" podID="74d86613-a3ce-4adf-bf86-934052abad09" containerID="187cc5b48080ec93b96673e51755e161ed8cc9cdc754c5580a00af09fc43a9b2" exitCode=0 Sep 06 05:59:52.462030 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:52.462006 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" event={"ID":"74d86613-a3ce-4adf-bf86-934052abad09","Type":"ContainerDied","Data":"187cc5b48080ec93b96673e51755e161ed8cc9cdc754c5580a00af09fc43a9b2"} Sep 06 05:59:52.462470 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:52.462281 2578 scope.go:117] "RemoveContainer" containerID="187cc5b48080ec93b96673e51755e161ed8cc9cdc754c5580a00af09fc43a9b2" Sep 06 05:59:53.467733 ip-10-0-135-32 kubenswrapper[2578]: I0906 05:59:53.467702 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-ff6vt" event={"ID":"74d86613-a3ce-4adf-bf86-934052abad09","Type":"ContainerStarted","Data":"b96d05af2d23efd2ce2a425ceb256d7c6412b894bb048119562a3e9f18e463ad"} Sep 06 06:00:08.985087 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:08.985046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 06:00:08.987417 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:08.987395 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36acd55-4ce4-4bdb-9518-cca261651a19-metrics-certs\") pod \"network-metrics-daemon-w74pq\" (UID: \"f36acd55-4ce4-4bdb-9518-cca261651a19\") " pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 06:00:09.142161 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:09.142131 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9rnw9\"" Sep 06 06:00:09.150102 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:09.150085 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w74pq" Sep 06 06:00:09.482316 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:09.482292 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w74pq"] Sep 06 06:00:09.484671 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:00:09.484629 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36acd55_4ce4_4bdb_9518_cca261651a19.slice/crio-31f89590977792dd683d03ed5d187711ac405fad1e807d0a0f2b2d36966bc896 WatchSource:0}: Error finding container 31f89590977792dd683d03ed5d187711ac405fad1e807d0a0f2b2d36966bc896: Status 404 returned error can't find the container with id 31f89590977792dd683d03ed5d187711ac405fad1e807d0a0f2b2d36966bc896 Sep 06 06:00:09.527346 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:09.527316 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w74pq" event={"ID":"f36acd55-4ce4-4bdb-9518-cca261651a19","Type":"ContainerStarted","Data":"31f89590977792dd683d03ed5d187711ac405fad1e807d0a0f2b2d36966bc896"} Sep 06 06:00:11.537011 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:11.536976 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w74pq" event={"ID":"f36acd55-4ce4-4bdb-9518-cca261651a19","Type":"ContainerStarted","Data":"d8a17a8837d2ad3e96cfa93360f8979d7104ab3b5fbf5187c9c4cfcfddcdfaa4"} Sep 06 06:00:11.537449 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:11.537029 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w74pq" event={"ID":"f36acd55-4ce4-4bdb-9518-cca261651a19","Type":"ContainerStarted","Data":"bceaf41bf5ac2087289ab3df5a0e37ab32c9c5e539d161781ad63b3e2965336f"} Sep 06 06:00:11.554431 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:11.554379 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w74pq" podStartSLOduration=375.048085726 podStartE2EDuration="6m16.554361266s" totalImagesPullingTime="1.50627554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 05:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:00:11.552737897 +0000 UTC m=+376.951068388" watchObservedRunningTime="2026-09-06 06:00:11.554361266 +0000 UTC m=+376.952691734" Sep 06 06:00:14.815771 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:14.815738 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 06:00:14.816203 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:14.816015 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="alertmanager" containerID="cri-o://a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409" gracePeriod=120 Sep 06 06:00:14.816203 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:14.816089 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="kube-rbac-proxy-metric" containerID="cri-o://fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867" gracePeriod=120 Sep 06 06:00:14.816203 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:14.816114 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="kube-rbac-proxy-web" containerID="cri-o://2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1" gracePeriod=120 Sep 06 06:00:14.816203 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:14.816149 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="prom-label-proxy" containerID="cri-o://12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3" gracePeriod=120 Sep 06 06:00:14.816203 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:14.816158 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="config-reloader" containerID="cri-o://d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82" gracePeriod=120 Sep 06 06:00:14.816477 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:14.816114 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="kube-rbac-proxy" containerID="cri-o://1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571" gracePeriod=120 Sep 06 06:00:15.555659 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:15.555627 2578 generic.go:356] "Generic (PLEG): container finished" podID="b8705aaf-1985-4467-913b-92faeb227d4b" containerID="12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3" exitCode=0 Sep 06 06:00:15.555659 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:15.555650 2578 generic.go:356] "Generic (PLEG): container finished" podID="b8705aaf-1985-4467-913b-92faeb227d4b" containerID="1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571" exitCode=0 Sep 06 06:00:15.555659 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:15.555656 2578 generic.go:356] "Generic (PLEG): container finished" podID="b8705aaf-1985-4467-913b-92faeb227d4b" containerID="d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82" exitCode=0 Sep 06 06:00:15.555659 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:15.555663 2578 generic.go:356] "Generic (PLEG): container finished" podID="b8705aaf-1985-4467-913b-92faeb227d4b" containerID="a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409" exitCode=0 Sep 06 06:00:15.555923 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:15.555694 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8705aaf-1985-4467-913b-92faeb227d4b","Type":"ContainerDied","Data":"12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3"} Sep 06 06:00:15.555923 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:15.555748 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8705aaf-1985-4467-913b-92faeb227d4b","Type":"ContainerDied","Data":"1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571"} Sep 06 06:00:15.555923 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:15.555773 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8705aaf-1985-4467-913b-92faeb227d4b","Type":"ContainerDied","Data":"d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82"} Sep 06 06:00:15.555923 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:15.555797 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8705aaf-1985-4467-913b-92faeb227d4b","Type":"ContainerDied","Data":"a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409"} Sep 06 06:00:16.100213 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.100187 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.247889 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.247803 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-main-tls\") pod \"b8705aaf-1985-4467-913b-92faeb227d4b\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " Sep 06 06:00:16.247889 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.247874 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-web-config\") pod \"b8705aaf-1985-4467-913b-92faeb227d4b\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " Sep 06 06:00:16.248127 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.247893 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy\") pod \"b8705aaf-1985-4467-913b-92faeb227d4b\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " Sep 06 06:00:16.248127 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.247931 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-config-volume\") pod \"b8705aaf-1985-4467-913b-92faeb227d4b\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " Sep 06 06:00:16.248127 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.247960 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b8705aaf-1985-4467-913b-92faeb227d4b-alertmanager-main-db\") pod \"b8705aaf-1985-4467-913b-92faeb227d4b\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " Sep 06 06:00:16.248127 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.247988 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8705aaf-1985-4467-913b-92faeb227d4b-tls-assets\") pod \"b8705aaf-1985-4467-913b-92faeb227d4b\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " Sep 06 06:00:16.248127 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.248014 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trtbv\" (UniqueName: \"kubernetes.io/projected/b8705aaf-1985-4467-913b-92faeb227d4b-kube-api-access-trtbv\") pod \"b8705aaf-1985-4467-913b-92faeb227d4b\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " Sep 06 06:00:16.248127 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.248037 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8705aaf-1985-4467-913b-92faeb227d4b-config-out\") pod \"b8705aaf-1985-4467-913b-92faeb227d4b\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " Sep 06 06:00:16.248127 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.248065 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8705aaf-1985-4467-913b-92faeb227d4b-metrics-client-ca\") pod \"b8705aaf-1985-4467-913b-92faeb227d4b\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " Sep 06 06:00:16.248127 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.248105 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-cluster-tls-config\") pod \"b8705aaf-1985-4467-913b-92faeb227d4b\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " Sep 06 06:00:16.248519 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.248145 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8705aaf-1985-4467-913b-92faeb227d4b-alertmanager-trusted-ca-bundle\") pod \"b8705aaf-1985-4467-913b-92faeb227d4b\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " Sep 06 06:00:16.248519 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.248174 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b8705aaf-1985-4467-913b-92faeb227d4b\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " Sep 06 06:00:16.248519 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.248203 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy-web\") pod \"b8705aaf-1985-4467-913b-92faeb227d4b\" (UID: \"b8705aaf-1985-4467-913b-92faeb227d4b\") " Sep 06 06:00:16.248519 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.248312 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8705aaf-1985-4467-913b-92faeb227d4b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b8705aaf-1985-4467-913b-92faeb227d4b" (UID: "b8705aaf-1985-4467-913b-92faeb227d4b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:00:16.248877 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.248518 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b8705aaf-1985-4467-913b-92faeb227d4b-alertmanager-main-db\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:16.248877 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.248518 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8705aaf-1985-4467-913b-92faeb227d4b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b8705aaf-1985-4467-913b-92faeb227d4b" (UID: "b8705aaf-1985-4467-913b-92faeb227d4b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:00:16.249398 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.249356 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8705aaf-1985-4467-913b-92faeb227d4b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b8705aaf-1985-4467-913b-92faeb227d4b" (UID: "b8705aaf-1985-4467-913b-92faeb227d4b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:00:16.251027 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.250946 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8705aaf-1985-4467-913b-92faeb227d4b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b8705aaf-1985-4467-913b-92faeb227d4b" (UID: "b8705aaf-1985-4467-913b-92faeb227d4b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:00:16.251417 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.251375 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-config-volume" (OuterVolumeSpecName: "config-volume") pod "b8705aaf-1985-4467-913b-92faeb227d4b" (UID: "b8705aaf-1985-4467-913b-92faeb227d4b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:16.251767 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.251730 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b8705aaf-1985-4467-913b-92faeb227d4b" (UID: "b8705aaf-1985-4467-913b-92faeb227d4b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:16.251877 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.251789 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b8705aaf-1985-4467-913b-92faeb227d4b" (UID: "b8705aaf-1985-4467-913b-92faeb227d4b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:16.252773 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.252738 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b8705aaf-1985-4467-913b-92faeb227d4b" (UID: "b8705aaf-1985-4467-913b-92faeb227d4b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:16.253059 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.253035 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8705aaf-1985-4467-913b-92faeb227d4b-config-out" (OuterVolumeSpecName: "config-out") pod "b8705aaf-1985-4467-913b-92faeb227d4b" (UID: "b8705aaf-1985-4467-913b-92faeb227d4b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:00:16.253124 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.253036 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b8705aaf-1985-4467-913b-92faeb227d4b" (UID: "b8705aaf-1985-4467-913b-92faeb227d4b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:16.253612 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.253589 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8705aaf-1985-4467-913b-92faeb227d4b-kube-api-access-trtbv" (OuterVolumeSpecName: "kube-api-access-trtbv") pod "b8705aaf-1985-4467-913b-92faeb227d4b" (UID: "b8705aaf-1985-4467-913b-92faeb227d4b"). InnerVolumeSpecName "kube-api-access-trtbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:00:16.256316 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.256174 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b8705aaf-1985-4467-913b-92faeb227d4b" (UID: "b8705aaf-1985-4467-913b-92faeb227d4b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:16.262622 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.262601 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-web-config" (OuterVolumeSpecName: "web-config") pod "b8705aaf-1985-4467-913b-92faeb227d4b" (UID: "b8705aaf-1985-4467-913b-92faeb227d4b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:16.349404 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.349372 2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8705aaf-1985-4467-913b-92faeb227d4b-tls-assets\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:16.349404 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.349402 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trtbv\" (UniqueName: \"kubernetes.io/projected/b8705aaf-1985-4467-913b-92faeb227d4b-kube-api-access-trtbv\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:16.349579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.349413 2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8705aaf-1985-4467-913b-92faeb227d4b-config-out\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:16.349579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.349421 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8705aaf-1985-4467-913b-92faeb227d4b-metrics-client-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:16.349579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.349429 2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-cluster-tls-config\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:16.349579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.349438 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8705aaf-1985-4467-913b-92faeb227d4b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:16.349579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.349447 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:16.349579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.349456 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:16.349579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.349464 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-main-tls\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:16.349579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.349471 2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-web-config\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:16.349579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.349480 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:16.349579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.349488 2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b8705aaf-1985-4467-913b-92faeb227d4b-config-volume\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:16.568593 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.568560 2578 generic.go:356] "Generic (PLEG): container finished" podID="b8705aaf-1985-4467-913b-92faeb227d4b" containerID="fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867" exitCode=0 Sep 06 06:00:16.568593 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.568588 2578 generic.go:356] "Generic (PLEG): container finished" podID="b8705aaf-1985-4467-913b-92faeb227d4b" containerID="2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1" exitCode=0 Sep 06 06:00:16.568788 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.568646 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.568788 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.568646 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8705aaf-1985-4467-913b-92faeb227d4b","Type":"ContainerDied","Data":"fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867"} Sep 06 06:00:16.568788 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.568777 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8705aaf-1985-4467-913b-92faeb227d4b","Type":"ContainerDied","Data":"2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1"} Sep 06 06:00:16.568886 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.568824 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b8705aaf-1985-4467-913b-92faeb227d4b","Type":"ContainerDied","Data":"44e9419e2b4ee876b0d0c4ad4f238920d167b44765a1829bf8d989133682c51b"} Sep 06 06:00:16.568886 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.568840 2578 scope.go:117] "RemoveContainer" containerID="12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3" Sep 06 06:00:16.576762 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.576739 2578 scope.go:117] "RemoveContainer" containerID="fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867" Sep 06 06:00:16.584149 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.584132 2578 scope.go:117] "RemoveContainer" containerID="1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571" Sep 06 06:00:16.590925 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.590889 2578 scope.go:117] "RemoveContainer" containerID="2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1" Sep 06 06:00:16.592450 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.592431 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 06:00:16.598265 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.598250 2578 scope.go:117] "RemoveContainer" containerID="d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82" Sep 06 06:00:16.598808 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.598791 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 06:00:16.605056 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.605040 2578 scope.go:117] "RemoveContainer" containerID="a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409" Sep 06 06:00:16.612041 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.612020 2578 scope.go:117] "RemoveContainer" containerID="6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1" Sep 06 06:00:16.619055 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.619038 2578 scope.go:117] "RemoveContainer" containerID="12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3" Sep 06 06:00:16.619322 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:00:16.619294 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3\": container with ID starting with 12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3 not found: ID does not exist" containerID="12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3" Sep 06 06:00:16.619360 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.619333 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3"} err="failed to get container status \"12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3\": rpc error: code = NotFound desc = could not find container \"12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3\": container with ID starting with 12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3 not found: ID does not exist" Sep 06 06:00:16.619360 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.619351 2578 scope.go:117] "RemoveContainer" containerID="fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867" Sep 06 06:00:16.619582 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:00:16.619566 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867\": container with ID starting with fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867 not found: ID does not exist" containerID="fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867" Sep 06 06:00:16.619628 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.619588 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867"} err="failed to get container status \"fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867\": rpc error: code = NotFound desc = could not find container \"fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867\": container with ID starting with fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867 not found: ID does not exist" Sep 06 06:00:16.619628 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.619603 2578 scope.go:117] "RemoveContainer" containerID="1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571" Sep 06 06:00:16.619850 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:00:16.619833 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571\": container with ID starting with 1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571 not found: ID does not exist" containerID="1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571" Sep 06 06:00:16.619930 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.619854 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571"} err="failed to get container status \"1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571\": rpc error: code = NotFound desc = could not find container \"1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571\": container with ID starting with 1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571 not found: ID does not exist" Sep 06 06:00:16.619930 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.619867 2578 scope.go:117] "RemoveContainer" containerID="2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1" Sep 06 06:00:16.620087 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:00:16.620072 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1\": container with ID starting with 2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1 not found: ID does not exist" containerID="2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1" Sep 06 06:00:16.620122 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.620090 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1"} err="failed to get container status \"2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1\": rpc error: code = NotFound desc = could not find container \"2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1\": container with ID starting with 2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1 not found: ID does not exist" Sep 06 06:00:16.620122 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.620101 2578 scope.go:117] "RemoveContainer" containerID="d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82" Sep 06 06:00:16.620298 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:00:16.620283 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82\": container with ID starting with d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82 not found: ID does not exist" containerID="d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82" Sep 06 06:00:16.620336 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.620301 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82"} err="failed to get container status \"d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82\": rpc error: code = NotFound desc = could not find container \"d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82\": container with ID starting with d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82 not found: ID does not exist" Sep 06 06:00:16.620336 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.620313 2578 scope.go:117] "RemoveContainer" containerID="a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409" Sep 06 06:00:16.620496 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:00:16.620481 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409\": container with ID starting with a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409 not found: ID does not exist" containerID="a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409" Sep 06 06:00:16.620531 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.620498 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409"} err="failed to get container status \"a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409\": rpc error: code = NotFound desc = could not find container \"a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409\": container with ID starting with a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409 not found: ID does not exist" Sep 06 06:00:16.620531 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.620508 2578 scope.go:117] "RemoveContainer" containerID="6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1" Sep 06 06:00:16.620694 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:00:16.620680 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1\": container with ID starting with 6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1 not found: ID does not exist" containerID="6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1" Sep 06 06:00:16.620735 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.620696 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1"} err="failed to get container status \"6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1\": rpc error: code = NotFound desc = could not find container \"6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1\": container with ID starting with 6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1 not found: ID does not exist" Sep 06 06:00:16.620735 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.620708 2578 scope.go:117] "RemoveContainer" containerID="12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3" Sep 06 06:00:16.620933 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.620896 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3"} err="failed to get container status \"12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3\": rpc error: code = NotFound desc = could not find container \"12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3\": container with ID starting with 12a72b6ab6f71be3106c91c25c31d584ea4e803d1c3a381c46da8830aa926ce3 not found: ID does not exist" Sep 06 06:00:16.620996 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.620936 2578 scope.go:117] "RemoveContainer" containerID="fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867" Sep 06 06:00:16.621177 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.621160 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867"} err="failed to get container status \"fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867\": rpc error: code = NotFound desc = could not find container \"fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867\": container with ID starting with fc5a21b1e2afcd20bbd7ff5224ee790132ea017f730e5c0efa17ad7294146867 not found: ID does not exist" Sep 06 06:00:16.621177 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.621176 2578 scope.go:117] "RemoveContainer" containerID="1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571" Sep 06 06:00:16.621404 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.621384 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571"} err="failed to get container status \"1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571\": rpc error: code = NotFound desc = could not find container \"1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571\": container with ID starting with 1200ebe74387b51795873de8ff9e35d9a6a12361580db168c2622e4db3a44571 not found: ID does not exist" Sep 06 06:00:16.621481 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.621407 2578 scope.go:117] "RemoveContainer" containerID="2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1" Sep 06 06:00:16.621633 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.621616 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1"} err="failed to get container status \"2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1\": rpc error: code = NotFound desc = could not find container \"2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1\": container with ID starting with 2a8dfab1c4263cc51b40dd1425032092c25c9983c5627328291bef127446d6c1 not found: ID does not exist" Sep 06 06:00:16.621702 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.621635 2578 scope.go:117] "RemoveContainer" containerID="d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82" Sep 06 06:00:16.621859 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.621837 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82"} err="failed to get container status \"d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82\": rpc error: code = NotFound desc = could not find container \"d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82\": container with ID starting with d11b2315167d889c59639c64bb67e24ac5325d5740e9d83eaf8309a8b0043d82 not found: ID does not exist" Sep 06 06:00:16.621859 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.621858 2578 scope.go:117] "RemoveContainer" containerID="a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409" Sep 06 06:00:16.622101 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.622085 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409"} err="failed to get container status \"a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409\": rpc error: code = NotFound desc = could not find container \"a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409\": container with ID starting with a51751e2b0c7e4186c47c182ecce6cb7849d978ec02344e00a584a1a2b311409 not found: ID does not exist" Sep 06 06:00:16.622146 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.622101 2578 scope.go:117] "RemoveContainer" containerID="6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1" Sep 06 06:00:16.622311 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.622296 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1"} err="failed to get container status \"6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1\": rpc error: code = NotFound desc = could not find container \"6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1\": container with ID starting with 6a5496d257eafa5ebc5c0f4cd811b44056360cb36fa5dcf7c4a30a3c6a6a6ca1 not found: ID does not exist" Sep 06 06:00:16.624539 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624521 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 06:00:16.624771 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624760 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="alertmanager" Sep 06 06:00:16.624812 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624773 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="alertmanager" Sep 06 06:00:16.624812 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624784 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="init-config-reloader" Sep 06 06:00:16.624812 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624790 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="init-config-reloader" Sep 06 06:00:16.624812 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624804 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="kube-rbac-proxy-web" Sep 06 06:00:16.624812 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624810 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="kube-rbac-proxy-web" Sep 06 06:00:16.624994 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624815 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="prom-label-proxy" Sep 06 06:00:16.624994 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624820 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="prom-label-proxy" Sep 06 06:00:16.624994 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624829 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="kube-rbac-proxy" Sep 06 06:00:16.624994 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624837 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="kube-rbac-proxy" Sep 06 06:00:16.624994 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624846 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="kube-rbac-proxy-metric" Sep 06 06:00:16.624994 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624854 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="kube-rbac-proxy-metric" Sep 06 06:00:16.624994 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624864 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="config-reloader" Sep 06 06:00:16.624994 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624872 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="config-reloader" Sep 06 06:00:16.624994 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.624994 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="config-reloader" Sep 06 06:00:16.625233 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.625004 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="kube-rbac-proxy" Sep 06 06:00:16.625233 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.625010 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="alertmanager" Sep 06 06:00:16.625233 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.625016 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="prom-label-proxy" Sep 06 06:00:16.625233 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.625022 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="kube-rbac-proxy-web" Sep 06 06:00:16.625233 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.625027 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" containerName="kube-rbac-proxy-metric" Sep 06 06:00:16.655461 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.655426 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 06:00:16.655613 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.655599 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.657989 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.657970 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 06 06:00:16.657989 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.657983 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 06 06:00:16.658237 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.658217 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 06 06:00:16.658582 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.658557 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 06 06:00:16.658675 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.658579 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 06 06:00:16.658781 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.658766 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 06 06:00:16.658843 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.658790 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zkwr2\"" Sep 06 06:00:16.658843 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.658796 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 06 06:00:16.658843 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.658807 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 06 06:00:16.663552 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.663534 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 06 06:00:16.752423 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.752392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fdb49626-f43e-4f1c-b4fc-28e68206df3a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.752423 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.752426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf6sq\" (UniqueName: \"kubernetes.io/projected/fdb49626-f43e-4f1c-b4fc-28e68206df3a-kube-api-access-hf6sq\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.752639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.752448 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.752639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.752475 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdb49626-f43e-4f1c-b4fc-28e68206df3a-config-out\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.752639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.752501 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.752639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.752533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.752639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.752557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49626-f43e-4f1c-b4fc-28e68206df3a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.752639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.752579 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-web-config\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.752639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.752634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-config-volume\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.752927 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.752668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.752927 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.752715 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.752927 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.752752 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fdb49626-f43e-4f1c-b4fc-28e68206df3a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.752927 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.752774 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb49626-f43e-4f1c-b4fc-28e68206df3a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.854055 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.853968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb49626-f43e-4f1c-b4fc-28e68206df3a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.854055 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.854012 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fdb49626-f43e-4f1c-b4fc-28e68206df3a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.854249 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.854134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf6sq\" (UniqueName: \"kubernetes.io/projected/fdb49626-f43e-4f1c-b4fc-28e68206df3a-kube-api-access-hf6sq\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.854249 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.854176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.854371 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.854348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdb49626-f43e-4f1c-b4fc-28e68206df3a-config-out\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.854432 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.854384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.854480 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.854432 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.854480 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.854463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49626-f43e-4f1c-b4fc-28e68206df3a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.854579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.854488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-web-config\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.854579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.854519 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-config-volume\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.854579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.854561 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.854719 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.854587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.854719 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.854634 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fdb49626-f43e-4f1c-b4fc-28e68206df3a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.854821 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.854800 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb49626-f43e-4f1c-b4fc-28e68206df3a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.854998 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.854977 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fdb49626-f43e-4f1c-b4fc-28e68206df3a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.855653 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.855624 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49626-f43e-4f1c-b4fc-28e68206df3a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.857470 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.857443 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.857470 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.857465 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdb49626-f43e-4f1c-b4fc-28e68206df3a-config-out\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.857653 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.857630 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.857739 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.857717 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.858059 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.858034 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-config-volume\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.858059 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.858055 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fdb49626-f43e-4f1c-b4fc-28e68206df3a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.858286 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.858266 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-web-config\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.858539 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.858517 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.859600 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.859580 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fdb49626-f43e-4f1c-b4fc-28e68206df3a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.861942 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.861926 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf6sq\" (UniqueName: \"kubernetes.io/projected/fdb49626-f43e-4f1c-b4fc-28e68206df3a-kube-api-access-hf6sq\") pod \"alertmanager-main-0\" (UID: \"fdb49626-f43e-4f1c-b4fc-28e68206df3a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:16.966316 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:16.966288 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 06 06:00:17.144479 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:17.144406 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8705aaf-1985-4467-913b-92faeb227d4b" path="/var/lib/kubelet/pods/b8705aaf-1985-4467-913b-92faeb227d4b/volumes" Sep 06 06:00:17.270263 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:17.270239 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 06 06:00:17.271718 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:00:17.271689 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb49626_f43e_4f1c_b4fc_28e68206df3a.slice/crio-c3d30f405f7ff09890788daead46632ec6487b5eec9bc6fbdf53f663f9c6521b WatchSource:0}: Error finding container c3d30f405f7ff09890788daead46632ec6487b5eec9bc6fbdf53f663f9c6521b: Status 404 returned error can't find the container with id c3d30f405f7ff09890788daead46632ec6487b5eec9bc6fbdf53f663f9c6521b Sep 06 06:00:17.575742 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:17.575708 2578 generic.go:356] "Generic (PLEG): container finished" podID="fdb49626-f43e-4f1c-b4fc-28e68206df3a" containerID="bae1a1b7ce69f0f6f80cb8e860be6e5870188891f39a260f5ccd21faa938f031" exitCode=0 Sep 06 06:00:17.575976 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:17.575789 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb49626-f43e-4f1c-b4fc-28e68206df3a","Type":"ContainerDied","Data":"bae1a1b7ce69f0f6f80cb8e860be6e5870188891f39a260f5ccd21faa938f031"} Sep 06 06:00:17.575976 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:17.575859 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb49626-f43e-4f1c-b4fc-28e68206df3a","Type":"ContainerStarted","Data":"c3d30f405f7ff09890788daead46632ec6487b5eec9bc6fbdf53f663f9c6521b"} Sep 06 06:00:18.583939 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.583888 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb49626-f43e-4f1c-b4fc-28e68206df3a","Type":"ContainerStarted","Data":"6709b2829c52ef3ee7380aff99d5264e7226d3fe8f83b73dee5c17b41a18e218"} Sep 06 06:00:18.584381 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.584020 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb49626-f43e-4f1c-b4fc-28e68206df3a","Type":"ContainerStarted","Data":"f55ff3794050f66e17f5f70d84f817f758ed0f89d8284d2ed52c56d40a8455e3"} Sep 06 06:00:18.584381 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.584062 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb49626-f43e-4f1c-b4fc-28e68206df3a","Type":"ContainerStarted","Data":"a9d28d739106314e9f2f7ad4d8e83104a72a5ec0745f3435b19a0a003993af72"} Sep 06 06:00:18.584381 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.584097 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb49626-f43e-4f1c-b4fc-28e68206df3a","Type":"ContainerStarted","Data":"830c1b9f42b0a2057d304a5a569d5eea1bb525dfb46d81f711c7c5192032eeea"} Sep 06 06:00:18.584381 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.584133 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb49626-f43e-4f1c-b4fc-28e68206df3a","Type":"ContainerStarted","Data":"df0f939d3f82251796f3c4c6ed80ab516519286385931847c3b451e62bc9b3db"} Sep 06 06:00:18.584381 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.584169 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fdb49626-f43e-4f1c-b4fc-28e68206df3a","Type":"ContainerStarted","Data":"bc434d1d3609691af465cebf05d0940d40fd70c19aa717850341f32e59d69a9c"} Sep 06 06:00:18.608807 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.608759 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.588733941 podStartE2EDuration="2.608742421s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.00848ms" isStatefulPod=false podCreationTimestamp="2026-09-06 06:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:00:18.606555147 +0000 UTC m=+384.004885634" watchObservedRunningTime="2026-09-06 06:00:18.608742421 +0000 UTC m=+384.007072888" Sep 06 06:00:18.820892 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.820849 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-864c4594ff-tq5nd"] Sep 06 06:00:18.846647 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.846564 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-864c4594ff-tq5nd"] Sep 06 06:00:18.846814 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.846695 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:18.848862 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.848830 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 06 06:00:18.849031 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.848946 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wmzsd\"" Sep 06 06:00:18.849031 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.849003 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 06 06:00:18.849031 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.848835 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 06 06:00:18.849216 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.849091 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 06 06:00:18.849417 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.849390 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 06 06:00:18.854266 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.854118 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 06 06:00:18.972870 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.972827 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28c1df2-0e0f-4985-b32e-861722441838-telemeter-trusted-ca-bundle\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:18.973061 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.972883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d28c1df2-0e0f-4985-b32e-861722441838-metrics-client-ca\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:18.973061 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.972980 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28c1df2-0e0f-4985-b32e-861722441838-serving-certs-ca-bundle\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:18.973061 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.973035 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d28c1df2-0e0f-4985-b32e-861722441838-secret-telemeter-client\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:18.973185 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.973069 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d28c1df2-0e0f-4985-b32e-861722441838-telemeter-client-tls\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:18.973185 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.973095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lklvr\" (UniqueName: \"kubernetes.io/projected/d28c1df2-0e0f-4985-b32e-861722441838-kube-api-access-lklvr\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:18.973185 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.973123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d28c1df2-0e0f-4985-b32e-861722441838-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:18.973316 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:18.973232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d28c1df2-0e0f-4985-b32e-861722441838-federate-client-tls\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.074173 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.074133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d28c1df2-0e0f-4985-b32e-861722441838-federate-client-tls\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.074173 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.074173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28c1df2-0e0f-4985-b32e-861722441838-telemeter-trusted-ca-bundle\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.074425 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.074195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d28c1df2-0e0f-4985-b32e-861722441838-metrics-client-ca\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.074425 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.074227 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28c1df2-0e0f-4985-b32e-861722441838-serving-certs-ca-bundle\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.074425 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.074290 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d28c1df2-0e0f-4985-b32e-861722441838-secret-telemeter-client\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.074425 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.074319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d28c1df2-0e0f-4985-b32e-861722441838-telemeter-client-tls\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.074425 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.074355 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lklvr\" (UniqueName: \"kubernetes.io/projected/d28c1df2-0e0f-4985-b32e-861722441838-kube-api-access-lklvr\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.074425 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.074386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d28c1df2-0e0f-4985-b32e-861722441838-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.075397 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.075366 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d28c1df2-0e0f-4985-b32e-861722441838-metrics-client-ca\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.075665 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.075641 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28c1df2-0e0f-4985-b32e-861722441838-telemeter-trusted-ca-bundle\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.075772 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.075741 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28c1df2-0e0f-4985-b32e-861722441838-serving-certs-ca-bundle\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.077175 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.077136 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d28c1df2-0e0f-4985-b32e-861722441838-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.077306 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.077152 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d28c1df2-0e0f-4985-b32e-861722441838-federate-client-tls\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.077306 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.077272 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d28c1df2-0e0f-4985-b32e-861722441838-secret-telemeter-client\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.077389 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.077348 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d28c1df2-0e0f-4985-b32e-861722441838-telemeter-client-tls\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.081256 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.081229 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lklvr\" (UniqueName: \"kubernetes.io/projected/d28c1df2-0e0f-4985-b32e-861722441838-kube-api-access-lklvr\") pod \"telemeter-client-864c4594ff-tq5nd\" (UID: \"d28c1df2-0e0f-4985-b32e-861722441838\") " pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.158712 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.158632 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" Sep 06 06:00:19.438717 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.438689 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-864c4594ff-tq5nd"] Sep 06 06:00:19.440218 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:00:19.440194 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28c1df2_0e0f_4985_b32e_861722441838.slice/crio-8e2824f78a814b4ab2e1387208c12085e577157035ae1c080cb88cc7107b88ae WatchSource:0}: Error finding container 8e2824f78a814b4ab2e1387208c12085e577157035ae1c080cb88cc7107b88ae: Status 404 returned error can't find the container with id 8e2824f78a814b4ab2e1387208c12085e577157035ae1c080cb88cc7107b88ae Sep 06 06:00:19.589011 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:19.588972 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" event={"ID":"d28c1df2-0e0f-4985-b32e-861722441838","Type":"ContainerStarted","Data":"8e2824f78a814b4ab2e1387208c12085e577157035ae1c080cb88cc7107b88ae"} Sep 06 06:00:21.604229 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:21.604143 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" event={"ID":"d28c1df2-0e0f-4985-b32e-861722441838","Type":"ContainerStarted","Data":"fb35d27ff1714bd9af35221124483848c221fcdfe2343e9d1b610f9e58d5666d"} Sep 06 06:00:21.604229 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:21.604203 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" event={"ID":"d28c1df2-0e0f-4985-b32e-861722441838","Type":"ContainerStarted","Data":"af5c16d50ee74101182192c4e347f916b9aa68de661b3d31281625c8423e1ade"} Sep 06 06:00:21.604229 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:21.604223 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" event={"ID":"d28c1df2-0e0f-4985-b32e-861722441838","Type":"ContainerStarted","Data":"ffee5f85bdf335305a8b86fde138996aadf04fc9d5d580a9ac640658c9b45cf2"} Sep 06 06:00:21.627058 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:21.627004 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-864c4594ff-tq5nd" podStartSLOduration=1.7532789229999999 podStartE2EDuration="3.626988654s" totalImagesPullingTime="1.873709731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:00:21.624712918 +0000 UTC m=+387.023043386" watchObservedRunningTime="2026-09-06 06:00:21.626988654 +0000 UTC m=+387.025319121" Sep 06 06:00:32.142377 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:32.142342 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-789c8d65c9-lf2f7"] Sep 06 06:00:57.164439 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.164376 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-789c8d65c9-lf2f7" podUID="e76934c1-092f-4826-900f-8a552f6eef21" containerName="console" containerID="cri-o://6d01ca78f2559a8498541002e34e78becc35bc61044404231a9c716bb483f060" gracePeriod=15 Sep 06 06:00:57.412082 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.412055 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 06:00:57.482712 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.482621 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e76934c1-092f-4826-900f-8a552f6eef21-console-serving-cert\") pod \"e76934c1-092f-4826-900f-8a552f6eef21\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " Sep 06 06:00:57.482712 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.482671 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-trusted-ca-bundle\") pod \"e76934c1-092f-4826-900f-8a552f6eef21\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " Sep 06 06:00:57.482712 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.482692 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-oauth-serving-cert\") pod \"e76934c1-092f-4826-900f-8a552f6eef21\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " Sep 06 06:00:57.482712 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.482712 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-service-ca\") pod \"e76934c1-092f-4826-900f-8a552f6eef21\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " Sep 06 06:00:57.483049 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.482756 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27lvx\" (UniqueName: \"kubernetes.io/projected/e76934c1-092f-4826-900f-8a552f6eef21-kube-api-access-27lvx\") pod \"e76934c1-092f-4826-900f-8a552f6eef21\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " Sep 06 06:00:57.483049 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.482797 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-console-config\") pod \"e76934c1-092f-4826-900f-8a552f6eef21\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " Sep 06 06:00:57.483049 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.482843 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e76934c1-092f-4826-900f-8a552f6eef21-console-oauth-config\") pod \"e76934c1-092f-4826-900f-8a552f6eef21\" (UID: \"e76934c1-092f-4826-900f-8a552f6eef21\") " Sep 06 06:00:57.483215 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.483114 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e76934c1-092f-4826-900f-8a552f6eef21" (UID: "e76934c1-092f-4826-900f-8a552f6eef21"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:00:57.483281 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.483205 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-service-ca" (OuterVolumeSpecName: "service-ca") pod "e76934c1-092f-4826-900f-8a552f6eef21" (UID: "e76934c1-092f-4826-900f-8a552f6eef21"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:00:57.483281 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.483217 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e76934c1-092f-4826-900f-8a552f6eef21" (UID: "e76934c1-092f-4826-900f-8a552f6eef21"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:00:57.483376 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.483308 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-console-config" (OuterVolumeSpecName: "console-config") pod "e76934c1-092f-4826-900f-8a552f6eef21" (UID: "e76934c1-092f-4826-900f-8a552f6eef21"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:00:57.485127 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.485101 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76934c1-092f-4826-900f-8a552f6eef21-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e76934c1-092f-4826-900f-8a552f6eef21" (UID: "e76934c1-092f-4826-900f-8a552f6eef21"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:57.485217 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.485197 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76934c1-092f-4826-900f-8a552f6eef21-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e76934c1-092f-4826-900f-8a552f6eef21" (UID: "e76934c1-092f-4826-900f-8a552f6eef21"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:00:57.485276 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.485255 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76934c1-092f-4826-900f-8a552f6eef21-kube-api-access-27lvx" (OuterVolumeSpecName: "kube-api-access-27lvx") pod "e76934c1-092f-4826-900f-8a552f6eef21" (UID: "e76934c1-092f-4826-900f-8a552f6eef21"). InnerVolumeSpecName "kube-api-access-27lvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:00:57.584186 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.584150 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e76934c1-092f-4826-900f-8a552f6eef21-console-serving-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:57.584186 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.584177 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-trusted-ca-bundle\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:57.584186 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.584189 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-oauth-serving-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:57.584398 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.584199 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:57.584398 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.584207 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27lvx\" (UniqueName: \"kubernetes.io/projected/e76934c1-092f-4826-900f-8a552f6eef21-kube-api-access-27lvx\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:57.584398 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.584216 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e76934c1-092f-4826-900f-8a552f6eef21-console-config\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:57.584398 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.584224 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e76934c1-092f-4826-900f-8a552f6eef21-console-oauth-config\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:00:57.743528 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.743443 2578 generic.go:356] "Generic (PLEG): container finished" podID="e76934c1-092f-4826-900f-8a552f6eef21" containerID="6d01ca78f2559a8498541002e34e78becc35bc61044404231a9c716bb483f060" exitCode=0 Sep 06 06:00:57.743528 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.743491 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-789c8d65c9-lf2f7" Sep 06 06:00:57.743528 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.743500 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-789c8d65c9-lf2f7" event={"ID":"e76934c1-092f-4826-900f-8a552f6eef21","Type":"ContainerDied","Data":"6d01ca78f2559a8498541002e34e78becc35bc61044404231a9c716bb483f060"} Sep 06 06:00:57.743738 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.743550 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-789c8d65c9-lf2f7" event={"ID":"e76934c1-092f-4826-900f-8a552f6eef21","Type":"ContainerDied","Data":"62fc6f7fe24259fb3cdc99c47bf0c7f4725dfde925b444275c5607262583cab1"} Sep 06 06:00:57.743738 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.743578 2578 scope.go:117] "RemoveContainer" containerID="6d01ca78f2559a8498541002e34e78becc35bc61044404231a9c716bb483f060" Sep 06 06:00:57.752319 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.752299 2578 scope.go:117] "RemoveContainer" containerID="6d01ca78f2559a8498541002e34e78becc35bc61044404231a9c716bb483f060" Sep 06 06:00:57.752587 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:00:57.752568 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d01ca78f2559a8498541002e34e78becc35bc61044404231a9c716bb483f060\": container with ID starting with 6d01ca78f2559a8498541002e34e78becc35bc61044404231a9c716bb483f060 not found: ID does not exist" containerID="6d01ca78f2559a8498541002e34e78becc35bc61044404231a9c716bb483f060" Sep 06 06:00:57.752664 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.752595 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d01ca78f2559a8498541002e34e78becc35bc61044404231a9c716bb483f060"} err="failed to get container status \"6d01ca78f2559a8498541002e34e78becc35bc61044404231a9c716bb483f060\": rpc error: code = NotFound desc = could not find container \"6d01ca78f2559a8498541002e34e78becc35bc61044404231a9c716bb483f060\": container with ID starting with 6d01ca78f2559a8498541002e34e78becc35bc61044404231a9c716bb483f060 not found: ID does not exist" Sep 06 06:00:57.762643 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.762615 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-789c8d65c9-lf2f7"] Sep 06 06:00:57.766463 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:57.766444 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-789c8d65c9-lf2f7"] Sep 06 06:00:59.143314 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:00:59.143282 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e76934c1-092f-4826-900f-8a552f6eef21" path="/var/lib/kubelet/pods/e76934c1-092f-4826-900f-8a552f6eef21/volumes" Sep 06 06:02:16.185948 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.185885 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64fc6fbb77-r7bwg"] Sep 06 06:02:16.186464 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.186346 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e76934c1-092f-4826-900f-8a552f6eef21" containerName="console" Sep 06 06:02:16.186464 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.186366 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e76934c1-092f-4826-900f-8a552f6eef21" containerName="console" Sep 06 06:02:16.186464 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.186459 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e76934c1-092f-4826-900f-8a552f6eef21" containerName="console" Sep 06 06:02:16.192566 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.190717 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.193014 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.192993 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-whcwh\"" Sep 06 06:02:16.193162 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.193137 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 06 06:02:16.193789 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.193770 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 06 06:02:16.193922 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.193859 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 06 06:02:16.194032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.194011 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 06 06:02:16.194032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.194030 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 06 06:02:16.199656 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.199624 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64fc6fbb77-r7bwg"] Sep 06 06:02:16.200489 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.200466 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 06 06:02:16.268115 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.268080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-oauth-serving-cert\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.268115 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.268119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-service-ca\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.268313 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.268142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/efe3a3a8-952e-4e24-8321-e027d4737e91-console-serving-cert\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.268313 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.268202 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-trusted-ca-bundle\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.268313 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.268260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscvt\" (UniqueName: \"kubernetes.io/projected/efe3a3a8-952e-4e24-8321-e027d4737e91-kube-api-access-nscvt\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.268313 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.268280 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-console-config\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.268313 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.268303 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/efe3a3a8-952e-4e24-8321-e027d4737e91-console-oauth-config\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.369664 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.369613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-oauth-serving-cert\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.369664 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.369670 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-service-ca\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.369889 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.369783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/efe3a3a8-952e-4e24-8321-e027d4737e91-console-serving-cert\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.369889 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.369818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-trusted-ca-bundle\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.369889 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.369875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscvt\" (UniqueName: \"kubernetes.io/projected/efe3a3a8-952e-4e24-8321-e027d4737e91-kube-api-access-nscvt\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.370078 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.369896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-console-config\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.370078 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.369954 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/efe3a3a8-952e-4e24-8321-e027d4737e91-console-oauth-config\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.370455 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.370424 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-service-ca\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.370602 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.370464 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-oauth-serving-cert\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.370678 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.370610 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-console-config\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.370812 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.370792 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-trusted-ca-bundle\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.372433 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.372416 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/efe3a3a8-952e-4e24-8321-e027d4737e91-console-serving-cert\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.372576 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.372553 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/efe3a3a8-952e-4e24-8321-e027d4737e91-console-oauth-config\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.377938 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.377895 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscvt\" (UniqueName: \"kubernetes.io/projected/efe3a3a8-952e-4e24-8321-e027d4737e91-kube-api-access-nscvt\") pod \"console-64fc6fbb77-r7bwg\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.504288 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.504208 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:16.773246 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:16.773218 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64fc6fbb77-r7bwg"] Sep 06 06:02:16.776393 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:02:16.776369 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe3a3a8_952e_4e24_8321_e027d4737e91.slice/crio-575bf31165f0ff837e6669d40d18f5d7f85b9f67385869603ec216621b610d00 WatchSource:0}: Error finding container 575bf31165f0ff837e6669d40d18f5d7f85b9f67385869603ec216621b610d00: Status 404 returned error can't find the container with id 575bf31165f0ff837e6669d40d18f5d7f85b9f67385869603ec216621b610d00 Sep 06 06:02:17.047733 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:17.047639 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64fc6fbb77-r7bwg" event={"ID":"efe3a3a8-952e-4e24-8321-e027d4737e91","Type":"ContainerStarted","Data":"e1e5803e6688efcb2715ce5e4d1690b23a0969145e06c51b9103a7c48cd6f656"} Sep 06 06:02:17.047733 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:17.047716 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64fc6fbb77-r7bwg" event={"ID":"efe3a3a8-952e-4e24-8321-e027d4737e91","Type":"ContainerStarted","Data":"575bf31165f0ff837e6669d40d18f5d7f85b9f67385869603ec216621b610d00"} Sep 06 06:02:17.064461 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:17.064414 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64fc6fbb77-r7bwg" podStartSLOduration=1.064395353 podStartE2EDuration="1.064395353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:02:17.063961679 +0000 UTC m=+502.462292148" watchObservedRunningTime="2026-09-06 06:02:17.064395353 +0000 UTC m=+502.462725825" Sep 06 06:02:26.505290 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:26.505247 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:26.505735 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:26.505320 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:26.510232 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:26.510209 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:02:27.092161 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:02:27.092131 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:03:30.307126 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:30.307091 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7glj5"] Sep 06 06:03:30.310260 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:30.310241 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7glj5" Sep 06 06:03:30.312602 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:30.312581 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-99czp\"" Sep 06 06:03:30.312689 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:30.312581 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 06 06:03:30.313315 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:30.313300 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 06 06:03:30.316867 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:30.316845 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7glj5"] Sep 06 06:03:30.352873 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:30.352841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjm9k\" (UniqueName: \"kubernetes.io/projected/a7852e0d-edcf-4956-bf0a-499ba71075f0-kube-api-access-cjm9k\") pod \"cert-manager-webhook-6486ffd997-7glj5\" (UID: \"a7852e0d-edcf-4956-bf0a-499ba71075f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7glj5" Sep 06 06:03:30.353094 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:30.352882 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7852e0d-edcf-4956-bf0a-499ba71075f0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7glj5\" (UID: \"a7852e0d-edcf-4956-bf0a-499ba71075f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7glj5" Sep 06 06:03:30.453521 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:30.453483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjm9k\" (UniqueName: \"kubernetes.io/projected/a7852e0d-edcf-4956-bf0a-499ba71075f0-kube-api-access-cjm9k\") pod \"cert-manager-webhook-6486ffd997-7glj5\" (UID: \"a7852e0d-edcf-4956-bf0a-499ba71075f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7glj5" Sep 06 06:03:30.453732 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:30.453528 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7852e0d-edcf-4956-bf0a-499ba71075f0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7glj5\" (UID: \"a7852e0d-edcf-4956-bf0a-499ba71075f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7glj5" Sep 06 06:03:30.461123 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:30.461091 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7852e0d-edcf-4956-bf0a-499ba71075f0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7glj5\" (UID: \"a7852e0d-edcf-4956-bf0a-499ba71075f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7glj5" Sep 06 06:03:30.461297 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:30.461277 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjm9k\" (UniqueName: \"kubernetes.io/projected/a7852e0d-edcf-4956-bf0a-499ba71075f0-kube-api-access-cjm9k\") pod \"cert-manager-webhook-6486ffd997-7glj5\" (UID: \"a7852e0d-edcf-4956-bf0a-499ba71075f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7glj5" Sep 06 06:03:30.619640 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:30.619535 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7glj5" Sep 06 06:03:30.896860 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:30.896783 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7glj5"] Sep 06 06:03:30.899914 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:03:30.899861 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7852e0d_edcf_4956_bf0a_499ba71075f0.slice/crio-2d9cf890009072888aa399a6e733cbad6f57b38aad4029fafd3cd1c46e433c8f WatchSource:0}: Error finding container 2d9cf890009072888aa399a6e733cbad6f57b38aad4029fafd3cd1c46e433c8f: Status 404 returned error can't find the container with id 2d9cf890009072888aa399a6e733cbad6f57b38aad4029fafd3cd1c46e433c8f Sep 06 06:03:31.332645 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:31.332610 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7glj5" event={"ID":"a7852e0d-edcf-4956-bf0a-499ba71075f0","Type":"ContainerStarted","Data":"2d9cf890009072888aa399a6e733cbad6f57b38aad4029fafd3cd1c46e433c8f"} Sep 06 06:03:34.348210 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:34.348168 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7glj5" event={"ID":"a7852e0d-edcf-4956-bf0a-499ba71075f0","Type":"ContainerStarted","Data":"7518c03a651735855509d21788c5a6842493daf334484dfc50110d4d26629765"} Sep 06 06:03:34.348210 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:34.348216 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7glj5" Sep 06 06:03:34.363004 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:34.362954 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-7glj5" podStartSLOduration=1.352521068 podStartE2EDuration="4.362935561s" totalImagesPullingTime="3.010414493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:03:34.361314776 +0000 UTC m=+579.759645244" watchObservedRunningTime="2026-09-06 06:03:34.362935561 +0000 UTC m=+579.761266030" Sep 06 06:03:39.576075 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:39.576035 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6k5b"] Sep 06 06:03:39.579236 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:39.579220 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6k5b" Sep 06 06:03:39.581343 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:39.581319 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 06 06:03:39.581931 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:39.581890 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 06 06:03:39.582026 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:39.582004 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-tfz9q\"" Sep 06 06:03:39.592268 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:39.592237 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6k5b"] Sep 06 06:03:39.630428 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:39.630387 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35cf5209-0bf5-426b-840d-8870ea811702-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z6k5b\" (UID: \"35cf5209-0bf5-426b-840d-8870ea811702\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6k5b" Sep 06 06:03:39.630596 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:39.630440 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dpjm\" (UniqueName: \"kubernetes.io/projected/35cf5209-0bf5-426b-840d-8870ea811702-kube-api-access-7dpjm\") pod \"openshift-lws-operator-fd8ccff4c-z6k5b\" (UID: \"35cf5209-0bf5-426b-840d-8870ea811702\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6k5b" Sep 06 06:03:39.731636 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:39.731605 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35cf5209-0bf5-426b-840d-8870ea811702-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z6k5b\" (UID: \"35cf5209-0bf5-426b-840d-8870ea811702\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6k5b" Sep 06 06:03:39.731789 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:39.731649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dpjm\" (UniqueName: \"kubernetes.io/projected/35cf5209-0bf5-426b-840d-8870ea811702-kube-api-access-7dpjm\") pod \"openshift-lws-operator-fd8ccff4c-z6k5b\" (UID: \"35cf5209-0bf5-426b-840d-8870ea811702\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6k5b" Sep 06 06:03:39.732028 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:39.732009 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35cf5209-0bf5-426b-840d-8870ea811702-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z6k5b\" (UID: \"35cf5209-0bf5-426b-840d-8870ea811702\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6k5b" Sep 06 06:03:39.739138 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:39.739116 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dpjm\" (UniqueName: \"kubernetes.io/projected/35cf5209-0bf5-426b-840d-8870ea811702-kube-api-access-7dpjm\") pod \"openshift-lws-operator-fd8ccff4c-z6k5b\" (UID: \"35cf5209-0bf5-426b-840d-8870ea811702\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6k5b" Sep 06 06:03:39.887996 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:39.887892 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6k5b" Sep 06 06:03:40.159725 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:40.159699 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6k5b"] Sep 06 06:03:40.161803 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:03:40.161775 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35cf5209_0bf5_426b_840d_8870ea811702.slice/crio-0a9e90aed79782d176aa85eea656c928b170eea9b6e75036f955a33c75c8313c WatchSource:0}: Error finding container 0a9e90aed79782d176aa85eea656c928b170eea9b6e75036f955a33c75c8313c: Status 404 returned error can't find the container with id 0a9e90aed79782d176aa85eea656c928b170eea9b6e75036f955a33c75c8313c Sep 06 06:03:40.354840 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:40.354810 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7glj5" Sep 06 06:03:40.373959 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:40.373922 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6k5b" event={"ID":"35cf5209-0bf5-426b-840d-8870ea811702","Type":"ContainerStarted","Data":"0a9e90aed79782d176aa85eea656c928b170eea9b6e75036f955a33c75c8313c"} Sep 06 06:03:43.387952 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:43.387894 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6k5b" event={"ID":"35cf5209-0bf5-426b-840d-8870ea811702","Type":"ContainerStarted","Data":"6d09fa0725fc7473124266fb686fb16463479fa489408400d5b43114c8ab3b83"} Sep 06 06:03:43.402513 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:43.402463 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6k5b" podStartSLOduration=1.355556821 podStartE2EDuration="4.402447041s" totalImagesPullingTime="3.04689022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:03:43.401235956 +0000 UTC m=+588.799566427" watchObservedRunningTime="2026-09-06 06:03:43.402447041 +0000 UTC m=+588.800777508" Sep 06 06:03:55.057624 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:55.057593 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 06:03:55.058096 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:55.057732 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 06:03:57.322758 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.322724 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n"] Sep 06 06:03:57.329554 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.329539 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:03:57.332575 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.332547 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 06 06:03:57.332575 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.332568 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 06 06:03:57.332748 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.332558 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4mmn2\"" Sep 06 06:03:57.332748 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.332590 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 06 06:03:57.335949 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.335925 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n"] Sep 06 06:03:57.490589 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.490544 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa-cert\") pod \"lws-controller-manager-6fb67bfd69-gss5n\" (UID: \"0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:03:57.490786 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.490613 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xjsp\" (UniqueName: \"kubernetes.io/projected/0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa-kube-api-access-4xjsp\") pod \"lws-controller-manager-6fb67bfd69-gss5n\" (UID: \"0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:03:57.490786 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.490648 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa-metrics-cert\") pod \"lws-controller-manager-6fb67bfd69-gss5n\" (UID: \"0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:03:57.490786 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.490674 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa-manager-config\") pod \"lws-controller-manager-6fb67bfd69-gss5n\" (UID: \"0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:03:57.591513 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.591410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa-cert\") pod \"lws-controller-manager-6fb67bfd69-gss5n\" (UID: \"0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:03:57.591513 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.591486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xjsp\" (UniqueName: \"kubernetes.io/projected/0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa-kube-api-access-4xjsp\") pod \"lws-controller-manager-6fb67bfd69-gss5n\" (UID: \"0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:03:57.591513 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.591510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa-metrics-cert\") pod \"lws-controller-manager-6fb67bfd69-gss5n\" (UID: \"0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:03:57.591804 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.591535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa-manager-config\") pod \"lws-controller-manager-6fb67bfd69-gss5n\" (UID: \"0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:03:57.592276 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.592201 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa-manager-config\") pod \"lws-controller-manager-6fb67bfd69-gss5n\" (UID: \"0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:03:57.594140 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.594115 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa-metrics-cert\") pod \"lws-controller-manager-6fb67bfd69-gss5n\" (UID: \"0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:03:57.594268 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.594249 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa-cert\") pod \"lws-controller-manager-6fb67bfd69-gss5n\" (UID: \"0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:03:57.599579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.599554 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xjsp\" (UniqueName: \"kubernetes.io/projected/0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa-kube-api-access-4xjsp\") pod \"lws-controller-manager-6fb67bfd69-gss5n\" (UID: \"0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa\") " pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:03:57.639434 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.639394 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:03:57.911654 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.911627 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n"] Sep 06 06:03:57.914138 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:03:57.914114 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c02d25d_d4c5_49c7_8a4a_d9c80f7105fa.slice/crio-ee5e0e606090eaa35cb13911a4696b50d7dcb660e5371849a17d8d35355f90db WatchSource:0}: Error finding container ee5e0e606090eaa35cb13911a4696b50d7dcb660e5371849a17d8d35355f90db: Status 404 returned error can't find the container with id ee5e0e606090eaa35cb13911a4696b50d7dcb660e5371849a17d8d35355f90db Sep 06 06:03:57.916319 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:57.916303 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 06:03:58.448463 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:03:58.448426 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" event={"ID":"0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa","Type":"ContainerStarted","Data":"ee5e0e606090eaa35cb13911a4696b50d7dcb660e5371849a17d8d35355f90db"} Sep 06 06:04:00.457973 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:00.457938 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" event={"ID":"0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa","Type":"ContainerStarted","Data":"643e4669202220221c09e89e385fbc53b274cf7a61ae6ebbc857933f5c9422da"} Sep 06 06:04:00.458375 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:00.458033 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:04:00.481123 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:00.481063 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" podStartSLOduration=1.558911149 podStartE2EDuration="3.481044162s" totalImagesPullingTime="1.922133013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:04:00.479438503 +0000 UTC m=+605.877768971" watchObservedRunningTime="2026-09-06 06:04:00.481044162 +0000 UTC m=+605.879374709" Sep 06 06:04:04.143396 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.143365 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c"] Sep 06 06:04:04.149278 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.149257 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.151534 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.151516 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 06 06:04:04.151890 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.151868 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cvg9l\"" Sep 06 06:04:04.152023 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.151877 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 06 06:04:04.152023 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.151894 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 06 06:04:04.152023 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.151922 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 06 06:04:04.162493 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.162464 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c"] Sep 06 06:04:04.251270 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.251230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b897cd959-rsx7c\" (UID: \"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.251270 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.251273 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266-manifests\") pod \"opendatahub-operator-controller-manager-b897cd959-rsx7c\" (UID: \"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.251505 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.251312 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266-tmp\") pod \"opendatahub-operator-controller-manager-b897cd959-rsx7c\" (UID: \"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.251505 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.251407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266-webhook-cert\") pod \"opendatahub-operator-controller-manager-b897cd959-rsx7c\" (UID: \"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.251505 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.251439 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrnc\" (UniqueName: \"kubernetes.io/projected/3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266-kube-api-access-zvrnc\") pod \"opendatahub-operator-controller-manager-b897cd959-rsx7c\" (UID: \"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.352336 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.352292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b897cd959-rsx7c\" (UID: \"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.352336 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.352334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266-manifests\") pod \"opendatahub-operator-controller-manager-b897cd959-rsx7c\" (UID: \"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.352580 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.352463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266-tmp\") pod \"opendatahub-operator-controller-manager-b897cd959-rsx7c\" (UID: \"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.352580 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.352518 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266-webhook-cert\") pod \"opendatahub-operator-controller-manager-b897cd959-rsx7c\" (UID: \"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.352580 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.352536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrnc\" (UniqueName: \"kubernetes.io/projected/3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266-kube-api-access-zvrnc\") pod \"opendatahub-operator-controller-manager-b897cd959-rsx7c\" (UID: \"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.352746 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.352681 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266-manifests\") pod \"opendatahub-operator-controller-manager-b897cd959-rsx7c\" (UID: \"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.355040 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.355014 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266-tmp\") pod \"opendatahub-operator-controller-manager-b897cd959-rsx7c\" (UID: \"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.355194 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.355174 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266-webhook-cert\") pod \"opendatahub-operator-controller-manager-b897cd959-rsx7c\" (UID: \"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.355284 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.355261 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b897cd959-rsx7c\" (UID: \"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.359754 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.359728 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrnc\" (UniqueName: \"kubernetes.io/projected/3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266-kube-api-access-zvrnc\") pod \"opendatahub-operator-controller-manager-b897cd959-rsx7c\" (UID: \"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266\") " pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.460504 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.460410 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:04.735586 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:04.735499 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c"] Sep 06 06:04:04.738651 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:04:04.738624 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0f2c6b_5ca7_45e5_8d6d_cce3eba84266.slice/crio-9451830f49a72fa2f7202b19d8cc128967ae21c7d527656607d65e91ce771f38 WatchSource:0}: Error finding container 9451830f49a72fa2f7202b19d8cc128967ae21c7d527656607d65e91ce771f38: Status 404 returned error can't find the container with id 9451830f49a72fa2f7202b19d8cc128967ae21c7d527656607d65e91ce771f38 Sep 06 06:04:05.484109 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:05.484071 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" event={"ID":"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266","Type":"ContainerStarted","Data":"9451830f49a72fa2f7202b19d8cc128967ae21c7d527656607d65e91ce771f38"} Sep 06 06:04:08.500403 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:08.500364 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" event={"ID":"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266","Type":"ContainerStarted","Data":"210749d36bc4a0ad58458a358c69d480c6f5c995426102af898abae5825a04de"} Sep 06 06:04:09.506647 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:09.506612 2578 generic.go:356] "Generic (PLEG): container finished" podID="3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266" containerID="210749d36bc4a0ad58458a358c69d480c6f5c995426102af898abae5825a04de" exitCode=0 Sep 06 06:04:09.507146 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:09.506698 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" event={"ID":"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266","Type":"ContainerDied","Data":"210749d36bc4a0ad58458a358c69d480c6f5c995426102af898abae5825a04de"} Sep 06 06:04:10.513702 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:10.513664 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" event={"ID":"3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266","Type":"ContainerStarted","Data":"d2b16cce7dc6a723653ee5e42d78704262994cb6eccd32846c4c5d49483bbd85"} Sep 06 06:04:10.514152 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:10.513805 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:10.535512 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:10.535466 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" podStartSLOduration=2.255244058 podStartE2EDuration="6.535450593s" totalImagesPullingTime="3.236477538s" totalInitContainerRuntime="1.043728997s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:04:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:04:10.534542595 +0000 UTC m=+615.932873074" watchObservedRunningTime="2026-09-06 06:04:10.535450593 +0000 UTC m=+615.933781140" Sep 06 06:04:11.464972 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:11.464895 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6fb67bfd69-gss5n" Sep 06 06:04:21.520820 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:21.520737 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b897cd959-rsx7c" Sep 06 06:04:29.365156 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.365116 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-649cdb6d6d-2b5p7"] Sep 06 06:04:29.368969 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.368940 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.379044 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.379014 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-649cdb6d6d-2b5p7"] Sep 06 06:04:29.466385 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.466346 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-console-serving-cert\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.466385 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.466392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7sqb\" (UniqueName: \"kubernetes.io/projected/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-kube-api-access-l7sqb\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.466639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.466420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-oauth-serving-cert\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.466639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.466480 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-trusted-ca-bundle\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.466639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.466521 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-console-config\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.466639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.466545 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-console-oauth-config\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.466639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.466571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-service-ca\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.567670 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.567627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7sqb\" (UniqueName: \"kubernetes.io/projected/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-kube-api-access-l7sqb\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.567670 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.567672 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-oauth-serving-cert\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.567940 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.567705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-trusted-ca-bundle\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.567940 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.567736 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-console-config\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.567940 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.567773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-console-oauth-config\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.568085 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.567953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-service-ca\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.568085 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.568052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-console-serving-cert\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.568529 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.568506 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-console-config\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.568620 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.568549 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-oauth-serving-cert\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.568724 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.568703 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-trusted-ca-bundle\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.568761 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.568711 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-service-ca\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.570489 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.570468 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-console-oauth-config\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.570718 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.570701 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-console-serving-cert\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.574601 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.574575 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7sqb\" (UniqueName: \"kubernetes.io/projected/1cbc8aa3-773f-4a89-9b2e-44f7b88dec42-kube-api-access-l7sqb\") pod \"console-649cdb6d6d-2b5p7\" (UID: \"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42\") " pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.680547 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.680458 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:29.960607 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:29.960577 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-649cdb6d6d-2b5p7"] Sep 06 06:04:29.963332 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:04:29.963298 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cbc8aa3_773f_4a89_9b2e_44f7b88dec42.slice/crio-201289617fa78fbbfd02a418e0045d06b44ddccdd261dd3ed469b3b29937e62e WatchSource:0}: Error finding container 201289617fa78fbbfd02a418e0045d06b44ddccdd261dd3ed469b3b29937e62e: Status 404 returned error can't find the container with id 201289617fa78fbbfd02a418e0045d06b44ddccdd261dd3ed469b3b29937e62e Sep 06 06:04:30.605268 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.605232 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-649cdb6d6d-2b5p7" event={"ID":"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42","Type":"ContainerStarted","Data":"9b14e3e3bf31382b2c0e02ff4e662f40ad0a7515a6b772557eeeeed8c5d25cc4"} Sep 06 06:04:30.605663 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.605306 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-649cdb6d6d-2b5p7" event={"ID":"1cbc8aa3-773f-4a89-9b2e-44f7b88dec42","Type":"ContainerStarted","Data":"201289617fa78fbbfd02a418e0045d06b44ddccdd261dd3ed469b3b29937e62e"} Sep 06 06:04:30.623091 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.623030 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-649cdb6d6d-2b5p7" podStartSLOduration=1.62301189 podStartE2EDuration="1.62301189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:04:30.622076962 +0000 UTC m=+636.020407441" watchObservedRunningTime="2026-09-06 06:04:30.62301189 +0000 UTC m=+636.021342358" Sep 06 06:04:30.867841 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.867759 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv"] Sep 06 06:04:30.872532 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.872512 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:30.874646 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.874623 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 06 06:04:30.874758 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.874661 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kzcwr\"" Sep 06 06:04:30.883582 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.883558 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv"] Sep 06 06:04:30.980731 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.980689 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9881c8ee-3e68-4e99-a17c-198451cbf660-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:30.980731 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.980731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9881c8ee-3e68-4e99-a17c-198451cbf660-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:30.980996 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.980764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9881c8ee-3e68-4e99-a17c-198451cbf660-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:30.980996 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.980785 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9881c8ee-3e68-4e99-a17c-198451cbf660-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:30.980996 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.980828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6pvl\" (UniqueName: \"kubernetes.io/projected/9881c8ee-3e68-4e99-a17c-198451cbf660-kube-api-access-c6pvl\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:30.980996 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.980865 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9881c8ee-3e68-4e99-a17c-198451cbf660-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:30.980996 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.980948 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9881c8ee-3e68-4e99-a17c-198451cbf660-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:30.980996 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.980984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9881c8ee-3e68-4e99-a17c-198451cbf660-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:30.980996 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:30.981000 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9881c8ee-3e68-4e99-a17c-198451cbf660-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.082107 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.082070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9881c8ee-3e68-4e99-a17c-198451cbf660-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.082320 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.082122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9881c8ee-3e68-4e99-a17c-198451cbf660-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.082320 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.082156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9881c8ee-3e68-4e99-a17c-198451cbf660-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.082320 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.082172 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9881c8ee-3e68-4e99-a17c-198451cbf660-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.082320 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.082196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9881c8ee-3e68-4e99-a17c-198451cbf660-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.082320 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.082254 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9881c8ee-3e68-4e99-a17c-198451cbf660-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.082320 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.082319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9881c8ee-3e68-4e99-a17c-198451cbf660-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.082631 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.082349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9881c8ee-3e68-4e99-a17c-198451cbf660-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.082631 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.082383 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6pvl\" (UniqueName: \"kubernetes.io/projected/9881c8ee-3e68-4e99-a17c-198451cbf660-kube-api-access-c6pvl\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.082738 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.082627 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9881c8ee-3e68-4e99-a17c-198451cbf660-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.082802 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.082783 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9881c8ee-3e68-4e99-a17c-198451cbf660-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.082853 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.082828 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9881c8ee-3e68-4e99-a17c-198451cbf660-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.082940 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.082863 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9881c8ee-3e68-4e99-a17c-198451cbf660-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.082995 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.082952 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9881c8ee-3e68-4e99-a17c-198451cbf660-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.084687 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.084664 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9881c8ee-3e68-4e99-a17c-198451cbf660-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.084942 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.084925 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9881c8ee-3e68-4e99-a17c-198451cbf660-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.089788 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.089758 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6pvl\" (UniqueName: \"kubernetes.io/projected/9881c8ee-3e68-4e99-a17c-198451cbf660-kube-api-access-c6pvl\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.090021 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.090005 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9881c8ee-3e68-4e99-a17c-198451cbf660-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv\" (UID: \"9881c8ee-3e68-4e99-a17c-198451cbf660\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.183823 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.183713 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:31.461047 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.461024 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv"] Sep 06 06:04:31.463549 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:04:31.463505 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9881c8ee_3e68_4e99_a17c_198451cbf660.slice/crio-40f5949c71845eafdce60f257d1e0daadbc5faa3985b4717a843cc0a53e79293 WatchSource:0}: Error finding container 40f5949c71845eafdce60f257d1e0daadbc5faa3985b4717a843cc0a53e79293: Status 404 returned error can't find the container with id 40f5949c71845eafdce60f257d1e0daadbc5faa3985b4717a843cc0a53e79293 Sep 06 06:04:31.610862 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:31.610825 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" event={"ID":"9881c8ee-3e68-4e99-a17c-198451cbf660","Type":"ContainerStarted","Data":"40f5949c71845eafdce60f257d1e0daadbc5faa3985b4717a843cc0a53e79293"} Sep 06 06:04:34.021409 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.021375 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xslf6"] Sep 06 06:04:34.028544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.028520 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-xslf6" Sep 06 06:04:34.030869 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.030829 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 06 06:04:34.031060 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.031039 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-cwckc\"" Sep 06 06:04:34.032643 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.032619 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xslf6"] Sep 06 06:04:34.114737 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.114701 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpwxw\" (UniqueName: \"kubernetes.io/projected/c525d127-0f22-4fc8-84e8-58a618a0dbaa-kube-api-access-qpwxw\") pod \"odh-model-controller-7998c67dc4-xslf6\" (UID: \"c525d127-0f22-4fc8-84e8-58a618a0dbaa\") " pod="opendatahub/odh-model-controller-7998c67dc4-xslf6" Sep 06 06:04:34.114953 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.114822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c525d127-0f22-4fc8-84e8-58a618a0dbaa-cert\") pod \"odh-model-controller-7998c67dc4-xslf6\" (UID: \"c525d127-0f22-4fc8-84e8-58a618a0dbaa\") " pod="opendatahub/odh-model-controller-7998c67dc4-xslf6" Sep 06 06:04:34.216299 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.216267 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpwxw\" (UniqueName: \"kubernetes.io/projected/c525d127-0f22-4fc8-84e8-58a618a0dbaa-kube-api-access-qpwxw\") pod \"odh-model-controller-7998c67dc4-xslf6\" (UID: \"c525d127-0f22-4fc8-84e8-58a618a0dbaa\") " pod="opendatahub/odh-model-controller-7998c67dc4-xslf6" Sep 06 06:04:34.216480 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.216329 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c525d127-0f22-4fc8-84e8-58a618a0dbaa-cert\") pod \"odh-model-controller-7998c67dc4-xslf6\" (UID: \"c525d127-0f22-4fc8-84e8-58a618a0dbaa\") " pod="opendatahub/odh-model-controller-7998c67dc4-xslf6" Sep 06 06:04:34.219148 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.219122 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c525d127-0f22-4fc8-84e8-58a618a0dbaa-cert\") pod \"odh-model-controller-7998c67dc4-xslf6\" (UID: \"c525d127-0f22-4fc8-84e8-58a618a0dbaa\") " pod="opendatahub/odh-model-controller-7998c67dc4-xslf6" Sep 06 06:04:34.223842 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.223816 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpwxw\" (UniqueName: \"kubernetes.io/projected/c525d127-0f22-4fc8-84e8-58a618a0dbaa-kube-api-access-qpwxw\") pod \"odh-model-controller-7998c67dc4-xslf6\" (UID: \"c525d127-0f22-4fc8-84e8-58a618a0dbaa\") " pod="opendatahub/odh-model-controller-7998c67dc4-xslf6" Sep 06 06:04:34.296568 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.296522 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:04:34.296667 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.296600 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:04:34.296667 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.296642 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:04:34.341267 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.341245 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-xslf6" Sep 06 06:04:34.615212 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.615040 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xslf6"] Sep 06 06:04:34.617888 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:04:34.617840 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc525d127_0f22_4fc8_84e8_58a618a0dbaa.slice/crio-435967742e21bb1bb9d2a32c59409d2b994dbb1f6053774e84f0b28060fc16bd WatchSource:0}: Error finding container 435967742e21bb1bb9d2a32c59409d2b994dbb1f6053774e84f0b28060fc16bd: Status 404 returned error can't find the container with id 435967742e21bb1bb9d2a32c59409d2b994dbb1f6053774e84f0b28060fc16bd Sep 06 06:04:34.628740 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.628700 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-xslf6" event={"ID":"c525d127-0f22-4fc8-84e8-58a618a0dbaa","Type":"ContainerStarted","Data":"435967742e21bb1bb9d2a32c59409d2b994dbb1f6053774e84f0b28060fc16bd"} Sep 06 06:04:34.630458 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.630424 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" event={"ID":"9881c8ee-3e68-4e99-a17c-198451cbf660","Type":"ContainerStarted","Data":"90cf1620e1720f2a278250a8379b86a92b2a964d2801ff84849618d40115f46f"} Sep 06 06:04:34.652390 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:34.652336 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" podStartSLOduration=1.821669038 podStartE2EDuration="4.652313144s" totalImagesPullingTime="2.830644106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:04:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:04:34.651693539 +0000 UTC m=+640.050024046" watchObservedRunningTime="2026-09-06 06:04:34.652313144 +0000 UTC m=+640.050643611" Sep 06 06:04:35.184385 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:35.184348 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:35.190571 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:35.190547 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:35.636280 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:35.636246 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:35.637523 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:35.637501 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv" Sep 06 06:04:37.648105 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:37.648019 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-xslf6" event={"ID":"c525d127-0f22-4fc8-84e8-58a618a0dbaa","Type":"ContainerStarted","Data":"ac30873413640f86a6a20b5c005ff233c72bfe1fe97c5a6bbb4db78ee9eec33a"} Sep 06 06:04:37.664038 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:37.663987 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-xslf6" podStartSLOduration=0.89534515 podStartE2EDuration="3.663971816s" totalImagesPullingTime="2.768626666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:04:37.661972732 +0000 UTC m=+643.060303219" watchObservedRunningTime="2026-09-06 06:04:37.663971816 +0000 UTC m=+643.062302283" Sep 06 06:04:38.652571 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:38.652540 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-xslf6" Sep 06 06:04:39.681087 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:39.681020 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:39.681552 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:39.681115 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:39.685838 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:39.685812 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:40.668249 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:40.668222 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-649cdb6d6d-2b5p7" Sep 06 06:04:40.718757 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:40.718726 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64fc6fbb77-r7bwg"] Sep 06 06:04:43.655423 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:43.655385 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b"] Sep 06 06:04:43.664549 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:43.664526 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b" Sep 06 06:04:43.667460 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:43.667440 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-fwrst\"" Sep 06 06:04:43.668143 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:43.668126 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 06 06:04:43.682149 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:43.682123 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b"] Sep 06 06:04:43.810763 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:43.810723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63866f27-a454-4498-b1ab-14973b58a81e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ssg8b\" (UID: \"63866f27-a454-4498-b1ab-14973b58a81e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b" Sep 06 06:04:43.810961 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:43.810769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wttjr\" (UniqueName: \"kubernetes.io/projected/63866f27-a454-4498-b1ab-14973b58a81e-kube-api-access-wttjr\") pod \"llmisvc-controller-manager-777c6bd7d5-ssg8b\" (UID: \"63866f27-a454-4498-b1ab-14973b58a81e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b" Sep 06 06:04:43.911880 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:43.911785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63866f27-a454-4498-b1ab-14973b58a81e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ssg8b\" (UID: \"63866f27-a454-4498-b1ab-14973b58a81e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b" Sep 06 06:04:43.911880 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:43.911831 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wttjr\" (UniqueName: \"kubernetes.io/projected/63866f27-a454-4498-b1ab-14973b58a81e-kube-api-access-wttjr\") pod \"llmisvc-controller-manager-777c6bd7d5-ssg8b\" (UID: \"63866f27-a454-4498-b1ab-14973b58a81e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b" Sep 06 06:04:43.914510 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:43.914488 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63866f27-a454-4498-b1ab-14973b58a81e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ssg8b\" (UID: \"63866f27-a454-4498-b1ab-14973b58a81e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b" Sep 06 06:04:43.921298 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:43.921269 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wttjr\" (UniqueName: \"kubernetes.io/projected/63866f27-a454-4498-b1ab-14973b58a81e-kube-api-access-wttjr\") pod \"llmisvc-controller-manager-777c6bd7d5-ssg8b\" (UID: \"63866f27-a454-4498-b1ab-14973b58a81e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b" Sep 06 06:04:43.974225 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:43.974192 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b" Sep 06 06:04:44.245181 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:44.245151 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b"] Sep 06 06:04:44.247290 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:04:44.247265 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod63866f27_a454_4498_b1ab_14973b58a81e.slice/crio-4b82ade7bd660856827d640bff51eb8b89b9a88cac44749fcc06517167178c9c WatchSource:0}: Error finding container 4b82ade7bd660856827d640bff51eb8b89b9a88cac44749fcc06517167178c9c: Status 404 returned error can't find the container with id 4b82ade7bd660856827d640bff51eb8b89b9a88cac44749fcc06517167178c9c Sep 06 06:04:44.689183 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:44.689148 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b" event={"ID":"63866f27-a454-4498-b1ab-14973b58a81e","Type":"ContainerStarted","Data":"4b82ade7bd660856827d640bff51eb8b89b9a88cac44749fcc06517167178c9c"} Sep 06 06:04:47.705528 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:47.705487 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b" event={"ID":"63866f27-a454-4498-b1ab-14973b58a81e","Type":"ContainerStarted","Data":"25b4f23677ecffcd4dbcdfb5821d1d29d4671a442918f3ff8cd8012efdcbd758"} Sep 06 06:04:47.705945 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:47.705596 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b" Sep 06 06:04:47.742875 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:47.742815 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b" podStartSLOduration=2.320870022 podStartE2EDuration="4.742797853s" totalImagesPullingTime="2.421927831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:04:47.740842866 +0000 UTC m=+653.139173347" watchObservedRunningTime="2026-09-06 06:04:47.742797853 +0000 UTC m=+653.141128324" Sep 06 06:04:49.659704 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:04:49.659669 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-xslf6" Sep 06 06:05:05.740086 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:05.740046 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64fc6fbb77-r7bwg" podUID="efe3a3a8-952e-4e24-8321-e027d4737e91" containerName="console" containerID="cri-o://e1e5803e6688efcb2715ce5e4d1690b23a0969145e06c51b9103a7c48cd6f656" gracePeriod=15 Sep 06 06:05:05.985971 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:05.985948 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:05:06.113339 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.113302 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/efe3a3a8-952e-4e24-8321-e027d4737e91-console-oauth-config\") pod \"efe3a3a8-952e-4e24-8321-e027d4737e91\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " Sep 06 06:05:06.113517 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.113394 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-console-config\") pod \"efe3a3a8-952e-4e24-8321-e027d4737e91\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " Sep 06 06:05:06.113517 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.113443 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/efe3a3a8-952e-4e24-8321-e027d4737e91-console-serving-cert\") pod \"efe3a3a8-952e-4e24-8321-e027d4737e91\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " Sep 06 06:05:06.113517 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.113461 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nscvt\" (UniqueName: \"kubernetes.io/projected/efe3a3a8-952e-4e24-8321-e027d4737e91-kube-api-access-nscvt\") pod \"efe3a3a8-952e-4e24-8321-e027d4737e91\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " Sep 06 06:05:06.113517 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.113491 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-service-ca\") pod \"efe3a3a8-952e-4e24-8321-e027d4737e91\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " Sep 06 06:05:06.113719 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.113516 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-oauth-serving-cert\") pod \"efe3a3a8-952e-4e24-8321-e027d4737e91\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " Sep 06 06:05:06.113719 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.113542 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-trusted-ca-bundle\") pod \"efe3a3a8-952e-4e24-8321-e027d4737e91\" (UID: \"efe3a3a8-952e-4e24-8321-e027d4737e91\") " Sep 06 06:05:06.114098 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.114072 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "efe3a3a8-952e-4e24-8321-e027d4737e91" (UID: "efe3a3a8-952e-4e24-8321-e027d4737e91"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:05:06.114220 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.114103 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "efe3a3a8-952e-4e24-8321-e027d4737e91" (UID: "efe3a3a8-952e-4e24-8321-e027d4737e91"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:05:06.114220 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.114103 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-console-config" (OuterVolumeSpecName: "console-config") pod "efe3a3a8-952e-4e24-8321-e027d4737e91" (UID: "efe3a3a8-952e-4e24-8321-e027d4737e91"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:05:06.114294 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.114205 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-service-ca" (OuterVolumeSpecName: "service-ca") pod "efe3a3a8-952e-4e24-8321-e027d4737e91" (UID: "efe3a3a8-952e-4e24-8321-e027d4737e91"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:05:06.115598 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.115575 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe3a3a8-952e-4e24-8321-e027d4737e91-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "efe3a3a8-952e-4e24-8321-e027d4737e91" (UID: "efe3a3a8-952e-4e24-8321-e027d4737e91"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:05:06.116345 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.116315 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe3a3a8-952e-4e24-8321-e027d4737e91-kube-api-access-nscvt" (OuterVolumeSpecName: "kube-api-access-nscvt") pod "efe3a3a8-952e-4e24-8321-e027d4737e91" (UID: "efe3a3a8-952e-4e24-8321-e027d4737e91"). InnerVolumeSpecName "kube-api-access-nscvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:05:06.116426 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.116410 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe3a3a8-952e-4e24-8321-e027d4737e91-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "efe3a3a8-952e-4e24-8321-e027d4737e91" (UID: "efe3a3a8-952e-4e24-8321-e027d4737e91"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:05:06.215032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.214985 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/efe3a3a8-952e-4e24-8321-e027d4737e91-console-serving-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:05:06.215032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.215026 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nscvt\" (UniqueName: \"kubernetes.io/projected/efe3a3a8-952e-4e24-8321-e027d4737e91-kube-api-access-nscvt\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:05:06.215032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.215039 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:05:06.215032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.215051 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-oauth-serving-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:05:06.215304 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.215059 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-trusted-ca-bundle\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:05:06.215304 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.215068 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/efe3a3a8-952e-4e24-8321-e027d4737e91-console-oauth-config\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:05:06.215304 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.215077 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/efe3a3a8-952e-4e24-8321-e027d4737e91-console-config\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:05:06.792966 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.792927 2578 generic.go:356] "Generic (PLEG): container finished" podID="efe3a3a8-952e-4e24-8321-e027d4737e91" containerID="e1e5803e6688efcb2715ce5e4d1690b23a0969145e06c51b9103a7c48cd6f656" exitCode=0 Sep 06 06:05:06.792966 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.792963 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64fc6fbb77-r7bwg" Sep 06 06:05:06.793405 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.792999 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64fc6fbb77-r7bwg" event={"ID":"efe3a3a8-952e-4e24-8321-e027d4737e91","Type":"ContainerDied","Data":"e1e5803e6688efcb2715ce5e4d1690b23a0969145e06c51b9103a7c48cd6f656"} Sep 06 06:05:06.793405 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.793048 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64fc6fbb77-r7bwg" event={"ID":"efe3a3a8-952e-4e24-8321-e027d4737e91","Type":"ContainerDied","Data":"575bf31165f0ff837e6669d40d18f5d7f85b9f67385869603ec216621b610d00"} Sep 06 06:05:06.793405 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.793071 2578 scope.go:117] "RemoveContainer" containerID="e1e5803e6688efcb2715ce5e4d1690b23a0969145e06c51b9103a7c48cd6f656" Sep 06 06:05:06.802109 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.802090 2578 scope.go:117] "RemoveContainer" containerID="e1e5803e6688efcb2715ce5e4d1690b23a0969145e06c51b9103a7c48cd6f656" Sep 06 06:05:06.802402 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:05:06.802384 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e5803e6688efcb2715ce5e4d1690b23a0969145e06c51b9103a7c48cd6f656\": container with ID starting with e1e5803e6688efcb2715ce5e4d1690b23a0969145e06c51b9103a7c48cd6f656 not found: ID does not exist" containerID="e1e5803e6688efcb2715ce5e4d1690b23a0969145e06c51b9103a7c48cd6f656" Sep 06 06:05:06.802449 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.802412 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e5803e6688efcb2715ce5e4d1690b23a0969145e06c51b9103a7c48cd6f656"} err="failed to get container status \"e1e5803e6688efcb2715ce5e4d1690b23a0969145e06c51b9103a7c48cd6f656\": rpc error: code = NotFound desc = could not find container \"e1e5803e6688efcb2715ce5e4d1690b23a0969145e06c51b9103a7c48cd6f656\": container with ID starting with e1e5803e6688efcb2715ce5e4d1690b23a0969145e06c51b9103a7c48cd6f656 not found: ID does not exist" Sep 06 06:05:06.813297 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.813270 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64fc6fbb77-r7bwg"] Sep 06 06:05:06.817370 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:06.817349 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64fc6fbb77-r7bwg"] Sep 06 06:05:07.144646 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:07.144560 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efe3a3a8-952e-4e24-8321-e027d4737e91" path="/var/lib/kubelet/pods/efe3a3a8-952e-4e24-8321-e027d4737e91/volumes" Sep 06 06:05:18.713767 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:18.713735 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ssg8b" Sep 06 06:05:49.609973 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:49.607201 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz"] Sep 06 06:05:49.609973 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:49.608020 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe3a3a8-952e-4e24-8321-e027d4737e91" containerName="console" Sep 06 06:05:49.609973 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:49.608039 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe3a3a8-952e-4e24-8321-e027d4737e91" containerName="console" Sep 06 06:05:49.609973 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:49.608205 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe3a3a8-952e-4e24-8321-e027d4737e91" containerName="console" Sep 06 06:05:49.612273 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:49.612239 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" Sep 06 06:05:49.614664 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:49.614639 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 06 06:05:49.615190 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:49.615171 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-d26zn\"" Sep 06 06:05:49.615261 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:49.615200 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 06 06:05:49.620354 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:49.620334 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz"] Sep 06 06:05:49.692251 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:49.692220 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csv7d\" (UniqueName: \"kubernetes.io/projected/565d4e6e-eadf-4ef4-9cdc-02560831c321-kube-api-access-csv7d\") pod \"limitador-operator-controller-manager-6c549b6d85-cr2rz\" (UID: \"565d4e6e-eadf-4ef4-9cdc-02560831c321\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" Sep 06 06:05:49.793020 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:49.792991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csv7d\" (UniqueName: \"kubernetes.io/projected/565d4e6e-eadf-4ef4-9cdc-02560831c321-kube-api-access-csv7d\") pod \"limitador-operator-controller-manager-6c549b6d85-cr2rz\" (UID: \"565d4e6e-eadf-4ef4-9cdc-02560831c321\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" Sep 06 06:05:49.812796 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:49.812763 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csv7d\" (UniqueName: \"kubernetes.io/projected/565d4e6e-eadf-4ef4-9cdc-02560831c321-kube-api-access-csv7d\") pod \"limitador-operator-controller-manager-6c549b6d85-cr2rz\" (UID: \"565d4e6e-eadf-4ef4-9cdc-02560831c321\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" Sep 06 06:05:49.923311 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:49.923233 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" Sep 06 06:05:50.201328 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:50.201306 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz"] Sep 06 06:05:50.203789 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:05:50.203760 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565d4e6e_eadf_4ef4_9cdc_02560831c321.slice/crio-c1228ac33c1db72c1109b620dc71132787d15d9a6fd8d6c2a5473b026f27467e WatchSource:0}: Error finding container c1228ac33c1db72c1109b620dc71132787d15d9a6fd8d6c2a5473b026f27467e: Status 404 returned error can't find the container with id c1228ac33c1db72c1109b620dc71132787d15d9a6fd8d6c2a5473b026f27467e Sep 06 06:05:50.996115 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:50.996068 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" event={"ID":"565d4e6e-eadf-4ef4-9cdc-02560831c321","Type":"ContainerStarted","Data":"c1228ac33c1db72c1109b620dc71132787d15d9a6fd8d6c2a5473b026f27467e"} Sep 06 06:05:53.008873 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:53.008790 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" event={"ID":"565d4e6e-eadf-4ef4-9cdc-02560831c321","Type":"ContainerStarted","Data":"f691e5be4ffcd7647736d90263792cc74d02777d9561ac1897475776579b7dec"} Sep 06 06:05:53.009249 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:53.008922 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" Sep 06 06:05:53.025497 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:05:53.025448 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" podStartSLOduration=1.5718274220000001 podStartE2EDuration="4.025434975s" totalImagesPullingTime="2.453607553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:05:53.02494192 +0000 UTC m=+718.423272412" watchObservedRunningTime="2026-09-06 06:05:53.025434975 +0000 UTC m=+718.423765443" Sep 06 06:06:00.762504 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:00.762464 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2"] Sep 06 06:06:00.767157 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:00.767135 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2" Sep 06 06:06:00.769006 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:00.768979 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 06 06:06:00.769163 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:00.769067 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 06 06:06:00.769163 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:00.769136 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9w9zr\"" Sep 06 06:06:00.771547 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:00.771525 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2"] Sep 06 06:06:00.905044 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:00.905009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs2t2\" (UniqueName: \"kubernetes.io/projected/76d99e41-d9a6-4a88-8960-00637b5b6efa-kube-api-access-vs2t2\") pod \"kuadrant-console-plugin-598f86468b-zsfs2\" (UID: \"76d99e41-d9a6-4a88-8960-00637b5b6efa\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2" Sep 06 06:06:00.905210 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:00.905057 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/76d99e41-d9a6-4a88-8960-00637b5b6efa-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-zsfs2\" (UID: \"76d99e41-d9a6-4a88-8960-00637b5b6efa\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2" Sep 06 06:06:00.905210 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:00.905191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/76d99e41-d9a6-4a88-8960-00637b5b6efa-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-zsfs2\" (UID: \"76d99e41-d9a6-4a88-8960-00637b5b6efa\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2" Sep 06 06:06:01.005589 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:01.005547 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/76d99e41-d9a6-4a88-8960-00637b5b6efa-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-zsfs2\" (UID: \"76d99e41-d9a6-4a88-8960-00637b5b6efa\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2" Sep 06 06:06:01.005755 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:01.005595 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs2t2\" (UniqueName: \"kubernetes.io/projected/76d99e41-d9a6-4a88-8960-00637b5b6efa-kube-api-access-vs2t2\") pod \"kuadrant-console-plugin-598f86468b-zsfs2\" (UID: \"76d99e41-d9a6-4a88-8960-00637b5b6efa\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2" Sep 06 06:06:01.005755 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:01.005631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/76d99e41-d9a6-4a88-8960-00637b5b6efa-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-zsfs2\" (UID: \"76d99e41-d9a6-4a88-8960-00637b5b6efa\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2" Sep 06 06:06:01.006367 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:01.006349 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/76d99e41-d9a6-4a88-8960-00637b5b6efa-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-zsfs2\" (UID: \"76d99e41-d9a6-4a88-8960-00637b5b6efa\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2" Sep 06 06:06:01.008162 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:01.008137 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/76d99e41-d9a6-4a88-8960-00637b5b6efa-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-zsfs2\" (UID: \"76d99e41-d9a6-4a88-8960-00637b5b6efa\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2" Sep 06 06:06:01.012661 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:01.012610 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs2t2\" (UniqueName: \"kubernetes.io/projected/76d99e41-d9a6-4a88-8960-00637b5b6efa-kube-api-access-vs2t2\") pod \"kuadrant-console-plugin-598f86468b-zsfs2\" (UID: \"76d99e41-d9a6-4a88-8960-00637b5b6efa\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2" Sep 06 06:06:01.077381 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:01.077348 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2" Sep 06 06:06:01.352232 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:01.352127 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2"] Sep 06 06:06:01.354860 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:06:01.354833 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76d99e41_d9a6_4a88_8960_00637b5b6efa.slice/crio-0b55137c2b9b1b5dd9e56bb1fe3903e409910f4774d3acd8737f79edfc03026f WatchSource:0}: Error finding container 0b55137c2b9b1b5dd9e56bb1fe3903e409910f4774d3acd8737f79edfc03026f: Status 404 returned error can't find the container with id 0b55137c2b9b1b5dd9e56bb1fe3903e409910f4774d3acd8737f79edfc03026f Sep 06 06:06:02.053361 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:02.053290 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2" event={"ID":"76d99e41-d9a6-4a88-8960-00637b5b6efa","Type":"ContainerStarted","Data":"0b55137c2b9b1b5dd9e56bb1fe3903e409910f4774d3acd8737f79edfc03026f"} Sep 06 06:06:04.016490 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:04.016453 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" Sep 06 06:06:06.081384 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:06.081356 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2" event={"ID":"76d99e41-d9a6-4a88-8960-00637b5b6efa","Type":"ContainerStarted","Data":"0ede976c4096263711d8b434aedd5d803927e11032f60469510b86de2a9d5636"} Sep 06 06:06:06.096514 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:06.096467 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zsfs2" podStartSLOduration=1.782780935 podStartE2EDuration="6.096451202s" totalImagesPullingTime="4.313670267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:06:06.095181835 +0000 UTC m=+731.493512302" watchObservedRunningTime="2026-09-06 06:06:06.096451202 +0000 UTC m=+731.494781668" Sep 06 06:06:11.500061 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.500027 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz"] Sep 06 06:06:11.500532 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.500266 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" podUID="565d4e6e-eadf-4ef4-9cdc-02560831c321" containerName="manager" containerID="cri-o://f691e5be4ffcd7647736d90263792cc74d02777d9561ac1897475776579b7dec" gracePeriod=2 Sep 06 06:06:11.504571 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.504544 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz"] Sep 06 06:06:11.525285 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.525262 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-9kxg6"] Sep 06 06:06:11.525647 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.525633 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="565d4e6e-eadf-4ef4-9cdc-02560831c321" containerName="manager" Sep 06 06:06:11.525697 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.525650 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="565d4e6e-eadf-4ef4-9cdc-02560831c321" containerName="manager" Sep 06 06:06:11.525731 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.525711 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="565d4e6e-eadf-4ef4-9cdc-02560831c321" containerName="manager" Sep 06 06:06:11.528746 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.528724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-9kxg6" Sep 06 06:06:11.530797 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:06:11.530767 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-cr2rz\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="565d4e6e-eadf-4ef4-9cdc-02560831c321" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" Sep 06 06:06:11.539387 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.539361 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-9kxg6"] Sep 06 06:06:11.698475 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.698440 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqnqr\" (UniqueName: \"kubernetes.io/projected/3c20a402-67b1-447a-91cd-2ef5c2d39255-kube-api-access-gqnqr\") pod \"limitador-operator-controller-manager-6c549b6d85-9kxg6\" (UID: \"3c20a402-67b1-447a-91cd-2ef5c2d39255\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-9kxg6" Sep 06 06:06:11.755204 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.755146 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" Sep 06 06:06:11.757139 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:06:11.757117 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-cr2rz\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="565d4e6e-eadf-4ef4-9cdc-02560831c321" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" Sep 06 06:06:11.799566 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.799535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqnqr\" (UniqueName: \"kubernetes.io/projected/3c20a402-67b1-447a-91cd-2ef5c2d39255-kube-api-access-gqnqr\") pod \"limitador-operator-controller-manager-6c549b6d85-9kxg6\" (UID: \"3c20a402-67b1-447a-91cd-2ef5c2d39255\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-9kxg6" Sep 06 06:06:11.811016 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.810986 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqnqr\" (UniqueName: \"kubernetes.io/projected/3c20a402-67b1-447a-91cd-2ef5c2d39255-kube-api-access-gqnqr\") pod \"limitador-operator-controller-manager-6c549b6d85-9kxg6\" (UID: \"3c20a402-67b1-447a-91cd-2ef5c2d39255\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-9kxg6" Sep 06 06:06:11.867956 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.867920 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-9kxg6" Sep 06 06:06:11.899974 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.899939 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csv7d\" (UniqueName: \"kubernetes.io/projected/565d4e6e-eadf-4ef4-9cdc-02560831c321-kube-api-access-csv7d\") pod \"565d4e6e-eadf-4ef4-9cdc-02560831c321\" (UID: \"565d4e6e-eadf-4ef4-9cdc-02560831c321\") " Sep 06 06:06:11.902290 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:11.902260 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565d4e6e-eadf-4ef4-9cdc-02560831c321-kube-api-access-csv7d" (OuterVolumeSpecName: "kube-api-access-csv7d") pod "565d4e6e-eadf-4ef4-9cdc-02560831c321" (UID: "565d4e6e-eadf-4ef4-9cdc-02560831c321"). InnerVolumeSpecName "kube-api-access-csv7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:06:12.001210 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:12.001160 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csv7d\" (UniqueName: \"kubernetes.io/projected/565d4e6e-eadf-4ef4-9cdc-02560831c321-kube-api-access-csv7d\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:06:12.109815 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:12.109772 2578 generic.go:356] "Generic (PLEG): container finished" podID="565d4e6e-eadf-4ef4-9cdc-02560831c321" containerID="f691e5be4ffcd7647736d90263792cc74d02777d9561ac1897475776579b7dec" exitCode=0 Sep 06 06:06:12.109815 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:12.109819 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" Sep 06 06:06:12.110077 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:12.109879 2578 scope.go:117] "RemoveContainer" containerID="f691e5be4ffcd7647736d90263792cc74d02777d9561ac1897475776579b7dec" Sep 06 06:06:12.112687 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:06:12.112657 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-cr2rz\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="565d4e6e-eadf-4ef4-9cdc-02560831c321" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" Sep 06 06:06:12.118745 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:12.118728 2578 scope.go:117] "RemoveContainer" containerID="f691e5be4ffcd7647736d90263792cc74d02777d9561ac1897475776579b7dec" Sep 06 06:06:12.119133 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:06:12.119113 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f691e5be4ffcd7647736d90263792cc74d02777d9561ac1897475776579b7dec\": container with ID starting with f691e5be4ffcd7647736d90263792cc74d02777d9561ac1897475776579b7dec not found: ID does not exist" containerID="f691e5be4ffcd7647736d90263792cc74d02777d9561ac1897475776579b7dec" Sep 06 06:06:12.119199 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:12.119144 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f691e5be4ffcd7647736d90263792cc74d02777d9561ac1897475776579b7dec"} err="failed to get container status \"f691e5be4ffcd7647736d90263792cc74d02777d9561ac1897475776579b7dec\": rpc error: code = NotFound desc = could not find container \"f691e5be4ffcd7647736d90263792cc74d02777d9561ac1897475776579b7dec\": container with ID starting with f691e5be4ffcd7647736d90263792cc74d02777d9561ac1897475776579b7dec not found: ID does not exist" Sep 06 06:06:12.120962 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:06:12.120930 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-cr2rz\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="565d4e6e-eadf-4ef4-9cdc-02560831c321" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" Sep 06 06:06:12.150488 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:12.150356 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-9kxg6"] Sep 06 06:06:12.152801 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:06:12.152769 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c20a402_67b1_447a_91cd_2ef5c2d39255.slice/crio-500363e6839982ab65bcc19b44cf038d35e46b6d4d62174e65179f72618b5d99 WatchSource:0}: Error finding container 500363e6839982ab65bcc19b44cf038d35e46b6d4d62174e65179f72618b5d99: Status 404 returned error can't find the container with id 500363e6839982ab65bcc19b44cf038d35e46b6d4d62174e65179f72618b5d99 Sep 06 06:06:13.116855 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:13.116814 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-9kxg6" event={"ID":"3c20a402-67b1-447a-91cd-2ef5c2d39255","Type":"ContainerStarted","Data":"db9ff89ec6dae103ff7c4208fe972024818ff6cae5dd8d778f89575469b9f507"} Sep 06 06:06:13.117306 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:13.116870 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-9kxg6" event={"ID":"3c20a402-67b1-447a-91cd-2ef5c2d39255","Type":"ContainerStarted","Data":"500363e6839982ab65bcc19b44cf038d35e46b6d4d62174e65179f72618b5d99"} Sep 06 06:06:13.117306 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:13.117031 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-9kxg6" Sep 06 06:06:13.141996 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:13.141929 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-9kxg6" podStartSLOduration=2.141890849 podStartE2EDuration="2.141890849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:06:13.141047279 +0000 UTC m=+738.539377746" watchObservedRunningTime="2026-09-06 06:06:13.141890849 +0000 UTC m=+738.540221318" Sep 06 06:06:13.147644 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:06:13.147544 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-cr2rz\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="565d4e6e-eadf-4ef4-9cdc-02560831c321" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-cr2rz" Sep 06 06:06:13.150538 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:13.150512 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="565d4e6e-eadf-4ef4-9cdc-02560831c321" path="/var/lib/kubelet/pods/565d4e6e-eadf-4ef4-9cdc-02560831c321/volumes" Sep 06 06:06:24.125162 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:24.125131 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-9kxg6" Sep 06 06:06:40.906220 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:40.906185 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl"] Sep 06 06:06:40.931782 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:40.931753 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl"] Sep 06 06:06:40.931975 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:40.931885 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:40.933996 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:40.933974 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-j2wzn\"" Sep 06 06:06:41.065163 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.065132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhl8\" (UniqueName: \"kubernetes.io/projected/b1e56087-4754-48ce-9c1c-8f16b57c74b1-kube-api-access-hfhl8\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.065356 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.065182 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1e56087-4754-48ce-9c1c-8f16b57c74b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.065356 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.065219 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e56087-4754-48ce-9c1c-8f16b57c74b1-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.065356 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.065235 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1e56087-4754-48ce-9c1c-8f16b57c74b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.065356 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.065250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1e56087-4754-48ce-9c1c-8f16b57c74b1-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.065356 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.065287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e56087-4754-48ce-9c1c-8f16b57c74b1-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.065356 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.065305 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1e56087-4754-48ce-9c1c-8f16b57c74b1-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.065356 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.065322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1e56087-4754-48ce-9c1c-8f16b57c74b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.065356 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.065340 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1e56087-4754-48ce-9c1c-8f16b57c74b1-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.166536 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.166452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1e56087-4754-48ce-9c1c-8f16b57c74b1-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.166536 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.166500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhl8\" (UniqueName: \"kubernetes.io/projected/b1e56087-4754-48ce-9c1c-8f16b57c74b1-kube-api-access-hfhl8\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.166536 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.166533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1e56087-4754-48ce-9c1c-8f16b57c74b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.166819 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.166567 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e56087-4754-48ce-9c1c-8f16b57c74b1-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.166819 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.166583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1e56087-4754-48ce-9c1c-8f16b57c74b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.166819 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.166599 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1e56087-4754-48ce-9c1c-8f16b57c74b1-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.166819 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.166657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e56087-4754-48ce-9c1c-8f16b57c74b1-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.166819 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.166675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1e56087-4754-48ce-9c1c-8f16b57c74b1-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.166819 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.166698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1e56087-4754-48ce-9c1c-8f16b57c74b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.167153 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.167052 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e56087-4754-48ce-9c1c-8f16b57c74b1-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.167153 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.167122 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1e56087-4754-48ce-9c1c-8f16b57c74b1-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.167235 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.167187 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1e56087-4754-48ce-9c1c-8f16b57c74b1-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.167324 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.167238 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1e56087-4754-48ce-9c1c-8f16b57c74b1-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.167443 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.167423 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1e56087-4754-48ce-9c1c-8f16b57c74b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.169049 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.169026 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1e56087-4754-48ce-9c1c-8f16b57c74b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.169171 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.169153 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1e56087-4754-48ce-9c1c-8f16b57c74b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.176150 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.176102 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1e56087-4754-48ce-9c1c-8f16b57c74b1-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.176745 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.176726 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhl8\" (UniqueName: \"kubernetes.io/projected/b1e56087-4754-48ce-9c1c-8f16b57c74b1-kube-api-access-hfhl8\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rkjhl\" (UID: \"b1e56087-4754-48ce-9c1c-8f16b57c74b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.242832 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.242793 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:41.522957 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.522931 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl"] Sep 06 06:06:41.525055 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:06:41.525026 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e56087_4754_48ce_9c1c_8f16b57c74b1.slice/crio-b22a1bdf3abe43ab5b708cbc30dc7df126c4f2985bbe3c0cbe98b374f8aea006 WatchSource:0}: Error finding container b22a1bdf3abe43ab5b708cbc30dc7df126c4f2985bbe3c0cbe98b374f8aea006: Status 404 returned error can't find the container with id b22a1bdf3abe43ab5b708cbc30dc7df126c4f2985bbe3c0cbe98b374f8aea006 Sep 06 06:06:41.527268 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.527235 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:06:41.527358 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.527324 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:06:41.527358 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:41.527354 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:06:42.267366 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:42.267332 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" event={"ID":"b1e56087-4754-48ce-9c1c-8f16b57c74b1","Type":"ContainerStarted","Data":"a616165a5fc5b9b345df7e2798b57a92f07320fd6741a053df1d9b51a99cbd34"} Sep 06 06:06:42.267737 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:42.267387 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" event={"ID":"b1e56087-4754-48ce-9c1c-8f16b57c74b1","Type":"ContainerStarted","Data":"b22a1bdf3abe43ab5b708cbc30dc7df126c4f2985bbe3c0cbe98b374f8aea006"} Sep 06 06:06:42.285965 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:42.285889 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" podStartSLOduration=2.2858728 podStartE2EDuration="2.2858728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:06:42.284265988 +0000 UTC m=+767.682596469" watchObservedRunningTime="2026-09-06 06:06:42.2858728 +0000 UTC m=+767.684203268" Sep 06 06:06:43.243276 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.243239 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:43.248056 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.248032 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:43.272510 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.272489 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:43.273319 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.273299 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:06:43.637881 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.637841 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:06:43.641543 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.641522 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-hzz62" Sep 06 06:06:43.643362 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.643341 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 06 06:06:43.647565 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.647540 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:06:43.678618 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.678589 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:06:43.789221 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.789186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g2tp\" (UniqueName: \"kubernetes.io/projected/89f58454-2203-4916-9a9f-76c089e3dcba-kube-api-access-8g2tp\") pod \"limitador-limitador-85d9db4d5d-hzz62\" (UID: \"89f58454-2203-4916-9a9f-76c089e3dcba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-hzz62" Sep 06 06:06:43.789388 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.789250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/89f58454-2203-4916-9a9f-76c089e3dcba-config-file\") pod \"limitador-limitador-85d9db4d5d-hzz62\" (UID: \"89f58454-2203-4916-9a9f-76c089e3dcba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-hzz62" Sep 06 06:06:43.890612 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.890513 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/89f58454-2203-4916-9a9f-76c089e3dcba-config-file\") pod \"limitador-limitador-85d9db4d5d-hzz62\" (UID: \"89f58454-2203-4916-9a9f-76c089e3dcba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-hzz62" Sep 06 06:06:43.890612 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.890606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g2tp\" (UniqueName: \"kubernetes.io/projected/89f58454-2203-4916-9a9f-76c089e3dcba-kube-api-access-8g2tp\") pod \"limitador-limitador-85d9db4d5d-hzz62\" (UID: \"89f58454-2203-4916-9a9f-76c089e3dcba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-hzz62" Sep 06 06:06:43.891169 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.891148 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/89f58454-2203-4916-9a9f-76c089e3dcba-config-file\") pod \"limitador-limitador-85d9db4d5d-hzz62\" (UID: \"89f58454-2203-4916-9a9f-76c089e3dcba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-hzz62" Sep 06 06:06:43.898451 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.898431 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g2tp\" (UniqueName: \"kubernetes.io/projected/89f58454-2203-4916-9a9f-76c089e3dcba-kube-api-access-8g2tp\") pod \"limitador-limitador-85d9db4d5d-hzz62\" (UID: \"89f58454-2203-4916-9a9f-76c089e3dcba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-hzz62" Sep 06 06:06:43.953218 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:43.953183 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-hzz62" Sep 06 06:06:44.224116 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:44.224092 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:06:44.226731 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:06:44.226703 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f58454_2203_4916_9a9f_76c089e3dcba.slice/crio-5ff7ebd82f8297ca2a439b1418281613fe5448908feca6ec0c92935bdf5c8f6d WatchSource:0}: Error finding container 5ff7ebd82f8297ca2a439b1418281613fe5448908feca6ec0c92935bdf5c8f6d: Status 404 returned error can't find the container with id 5ff7ebd82f8297ca2a439b1418281613fe5448908feca6ec0c92935bdf5c8f6d Sep 06 06:06:44.278359 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:44.278329 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-hzz62" event={"ID":"89f58454-2203-4916-9a9f-76c089e3dcba","Type":"ContainerStarted","Data":"5ff7ebd82f8297ca2a439b1418281613fe5448908feca6ec0c92935bdf5c8f6d"} Sep 06 06:06:45.283716 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:45.283679 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-hzz62" event={"ID":"89f58454-2203-4916-9a9f-76c089e3dcba","Type":"ContainerStarted","Data":"118faa30c9260df8808680ffc0bf83cc8ffc3f9fcae647ae3f1ef04542cae8b8"} Sep 06 06:06:45.284150 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:45.283762 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-hzz62" Sep 06 06:06:45.299043 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:45.298988 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-hzz62" podStartSLOduration=1.393803631 podStartE2EDuration="2.298972047s" totalImagesPullingTime="905.168416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:06:45.297613353 +0000 UTC m=+770.695943838" watchObservedRunningTime="2026-09-06 06:06:45.298972047 +0000 UTC m=+770.697302514" Sep 06 06:06:56.290058 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:06:56.290025 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-hzz62" Sep 06 06:07:56.751895 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:07:56.751857 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:08:06.691639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:06.691606 2578 generic.go:356] "Generic (PLEG): container finished" podID="b1e56087-4754-48ce-9c1c-8f16b57c74b1" containerID="a616165a5fc5b9b345df7e2798b57a92f07320fd6741a053df1d9b51a99cbd34" exitCode=0 Sep 06 06:08:06.692061 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:06.691683 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" event={"ID":"b1e56087-4754-48ce-9c1c-8f16b57c74b1","Type":"ContainerDied","Data":"a616165a5fc5b9b345df7e2798b57a92f07320fd6741a053df1d9b51a99cbd34"} Sep 06 06:08:06.692110 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:06.692070 2578 scope.go:117] "RemoveContainer" containerID="a616165a5fc5b9b345df7e2798b57a92f07320fd6741a053df1d9b51a99cbd34" Sep 06 06:08:06.693068 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:06.693040 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:08:06.693173 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:06.693121 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:08:06.693240 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:06.693182 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:08:07.698119 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:07.698082 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" event={"ID":"b1e56087-4754-48ce-9c1c-8f16b57c74b1","Type":"ContainerStarted","Data":"12650faa33035ba102bbcbf0fd4717789a83559f4f1d9e01a63e856893d8c05c"} Sep 06 06:08:08.243493 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:08.243455 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:08:08.243662 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:08.243632 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rkjhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 06 06:08:08.243710 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:08.243675 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" podUID="b1e56087-4754-48ce-9c1c-8f16b57c74b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 06 06:08:09.243607 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:09.243568 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rkjhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 06 06:08:09.244026 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:09.243637 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" podUID="b1e56087-4754-48ce-9c1c-8f16b57c74b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 06 06:08:10.243553 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:10.243521 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rkjhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 06 06:08:10.243724 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:10.243591 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" podUID="b1e56087-4754-48ce-9c1c-8f16b57c74b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 06 06:08:11.243260 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:11.243222 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rkjhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 06 06:08:11.243435 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:11.243279 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" podUID="b1e56087-4754-48ce-9c1c-8f16b57c74b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 06 06:08:12.243540 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:12.243488 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rkjhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 06 06:08:12.243953 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:12.243570 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" podUID="b1e56087-4754-48ce-9c1c-8f16b57c74b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 06 06:08:13.247795 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:13.247765 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:08:13.248280 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:13.248147 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:08:13.248758 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:13.248741 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rkjhl" Sep 06 06:08:14.901595 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:14.901558 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:08:23.981009 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:23.980974 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:08:30.681592 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:30.681559 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:08:48.336918 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:48.336815 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:08:55.088725 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:55.088696 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 06:08:55.090252 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:08:55.090230 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 06:09:00.806397 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:00.806357 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6"] Sep 06 06:09:00.810354 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:00.810335 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:00.812659 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:00.812636 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 06 06:09:00.813381 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:00.813356 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nbtvj\"" Sep 06 06:09:00.813493 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:00.813415 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 06 06:09:00.813493 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:00.813421 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 06 06:09:00.818037 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:00.818011 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6"] Sep 06 06:09:00.939423 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:00.939390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba09e890-672a-49c5-a57a-c44d16e07c1d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:00.939594 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:00.939431 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zs9d\" (UniqueName: \"kubernetes.io/projected/ba09e890-672a-49c5-a57a-c44d16e07c1d-kube-api-access-6zs9d\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:00.939594 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:00.939459 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba09e890-672a-49c5-a57a-c44d16e07c1d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:00.939594 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:00.939520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba09e890-672a-49c5-a57a-c44d16e07c1d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:00.939594 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:00.939539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba09e890-672a-49c5-a57a-c44d16e07c1d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:00.939594 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:00.939558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba09e890-672a-49c5-a57a-c44d16e07c1d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:01.040761 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.040720 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba09e890-672a-49c5-a57a-c44d16e07c1d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:01.040761 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.040766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba09e890-672a-49c5-a57a-c44d16e07c1d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:01.041049 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.040795 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba09e890-672a-49c5-a57a-c44d16e07c1d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:01.041049 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.040889 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba09e890-672a-49c5-a57a-c44d16e07c1d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:01.041049 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.040948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zs9d\" (UniqueName: \"kubernetes.io/projected/ba09e890-672a-49c5-a57a-c44d16e07c1d-kube-api-access-6zs9d\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:01.041049 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.040989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba09e890-672a-49c5-a57a-c44d16e07c1d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:01.041269 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.041157 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba09e890-672a-49c5-a57a-c44d16e07c1d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:01.041269 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.041187 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba09e890-672a-49c5-a57a-c44d16e07c1d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:01.041378 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.041352 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba09e890-672a-49c5-a57a-c44d16e07c1d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:01.043190 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.043168 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba09e890-672a-49c5-a57a-c44d16e07c1d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:01.043483 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.043464 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba09e890-672a-49c5-a57a-c44d16e07c1d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:01.048500 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.048479 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zs9d\" (UniqueName: \"kubernetes.io/projected/ba09e890-672a-49c5-a57a-c44d16e07c1d-kube-api-access-6zs9d\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6\" (UID: \"ba09e890-672a-49c5-a57a-c44d16e07c1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:01.122414 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.122324 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:01.396871 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.396844 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6"] Sep 06 06:09:01.399462 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:09:01.399435 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba09e890_672a_49c5_a57a_c44d16e07c1d.slice/crio-58d0064dbdac757441342e13aa06d51266ef5709cec5036e69364df30df1ee87 WatchSource:0}: Error finding container 58d0064dbdac757441342e13aa06d51266ef5709cec5036e69364df30df1ee87: Status 404 returned error can't find the container with id 58d0064dbdac757441342e13aa06d51266ef5709cec5036e69364df30df1ee87 Sep 06 06:09:01.400714 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.400693 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 06:09:01.972465 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:01.972421 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" event={"ID":"ba09e890-672a-49c5-a57a-c44d16e07c1d","Type":"ContainerStarted","Data":"58d0064dbdac757441342e13aa06d51266ef5709cec5036e69364df30df1ee87"} Sep 06 06:09:02.187872 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:02.187840 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:09:02.979155 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:02.979106 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" event={"ID":"ba09e890-672a-49c5-a57a-c44d16e07c1d","Type":"ContainerStarted","Data":"1a7977faf11bdcd1e80b43bfa0621069db956f9dbaac516e8b2a0434cc3ec4da"} Sep 06 06:09:02.995753 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:02.995697 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" podStartSLOduration=2.024107124 podStartE2EDuration="2.995679928s" totalImagesPullingTime="971.572804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:09:02.993429469 +0000 UTC m=+908.391759949" watchObservedRunningTime="2026-09-06 06:09:02.995679928 +0000 UTC m=+908.394010396" Sep 06 06:09:06.886160 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:06.886121 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:09:11.123008 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:11.122973 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:11.123409 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:11.123048 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:11.135750 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:11.135720 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:12.033162 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:12.033133 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6" Sep 06 06:09:35.685585 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:09:35.685546 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:10:50.808418 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.808376 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm"] Sep 06 06:10:50.812072 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.812048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.814296 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.814267 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ceca3b-openshift-default-dockercfg-87zdv\"" Sep 06 06:10:50.821597 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.821573 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm"] Sep 06 06:10:50.836259 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.836222 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-data\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.836452 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.836276 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-workload-socket\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.836452 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.836293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-token\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.836452 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.836308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istiod-ca-cert\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.836452 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.836425 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-podinfo\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.836690 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.836454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-workload-certs\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.836690 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.836474 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-envoy\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.836690 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.836503 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s994\" (UniqueName: \"kubernetes.io/projected/cd5046cd-6f68-45bf-8e6b-59444f9af36d-kube-api-access-2s994\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.836690 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.836539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-credential-socket\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.937395 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.937362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-podinfo\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.937395 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.937398 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-workload-certs\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.937605 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.937413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-envoy\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.937605 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.937543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s994\" (UniqueName: \"kubernetes.io/projected/cd5046cd-6f68-45bf-8e6b-59444f9af36d-kube-api-access-2s994\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.937684 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.937615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-credential-socket\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.937684 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.937652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-data\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.937785 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.937722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-workload-socket\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.937839 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.937825 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-workload-certs\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.937940 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.937894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-token\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.938013 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.937959 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istiod-ca-cert\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.938064 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.938007 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-credential-socket\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.938106 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.938058 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-workload-socket\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.938374 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.938307 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-data\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.938575 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.938556 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istiod-ca-cert\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.940053 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.940030 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-envoy\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.940171 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.940125 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-podinfo\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.944703 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.944669 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-token\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:50.944819 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:50.944805 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s994\" (UniqueName: \"kubernetes.io/projected/cd5046cd-6f68-45bf-8e6b-59444f9af36d-kube-api-access-2s994\") pod \"e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:51.123380 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:51.123291 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:51.401633 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:51.401605 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm"] Sep 06 06:10:51.403839 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:10:51.403807 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5046cd_6f68_45bf_8e6b_59444f9af36d.slice/crio-f07e5d8ff5a54cb8bb5001e6de6bc3d7b3bf3c0ae8bdff697ef518e177c5395d WatchSource:0}: Error finding container f07e5d8ff5a54cb8bb5001e6de6bc3d7b3bf3c0ae8bdff697ef518e177c5395d: Status 404 returned error can't find the container with id f07e5d8ff5a54cb8bb5001e6de6bc3d7b3bf3c0ae8bdff697ef518e177c5395d Sep 06 06:10:51.406093 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:51.406061 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:10:51.406176 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:51.406134 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:10:51.406218 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:51.406173 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:10:51.506409 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:51.506365 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" event={"ID":"cd5046cd-6f68-45bf-8e6b-59444f9af36d","Type":"ContainerStarted","Data":"f86d4da5d4968e152942f0e3f0b37e044c235b256d31bb13d8a3d5312279afb9"} Sep 06 06:10:51.506576 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:51.506432 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" event={"ID":"cd5046cd-6f68-45bf-8e6b-59444f9af36d","Type":"ContainerStarted","Data":"f07e5d8ff5a54cb8bb5001e6de6bc3d7b3bf3c0ae8bdff697ef518e177c5395d"} Sep 06 06:10:51.524542 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:51.524491 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" podStartSLOduration=1.524472828 podStartE2EDuration="1.524472828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:10:51.522786367 +0000 UTC m=+1016.921116854" watchObservedRunningTime="2026-09-06 06:10:51.524472828 +0000 UTC m=+1016.922803295" Sep 06 06:10:52.124266 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:52.124231 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:52.130092 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:52.130056 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:52.435878 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:52.435793 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7737-140142029905712"] Sep 06 06:10:52.439436 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:52.439416 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-140142029905712" Sep 06 06:10:52.442465 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:52.442441 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-j266t\"" Sep 06 06:10:52.453337 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:52.452942 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7737-140142029905712"] Sep 06 06:10:52.512405 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:52.512374 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:52.513422 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:52.513403 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:10:52.554709 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:52.554672 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glqwj\" (UniqueName: \"kubernetes.io/projected/ef8526b2-45c8-48f7-9347-77bcdb8cb957-kube-api-access-glqwj\") pod \"test-curl-7737-140142029905712\" (UID: \"ef8526b2-45c8-48f7-9347-77bcdb8cb957\") " pod="opendatahub/test-curl-7737-140142029905712" Sep 06 06:10:52.656328 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:52.656294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glqwj\" (UniqueName: \"kubernetes.io/projected/ef8526b2-45c8-48f7-9347-77bcdb8cb957-kube-api-access-glqwj\") pod \"test-curl-7737-140142029905712\" (UID: \"ef8526b2-45c8-48f7-9347-77bcdb8cb957\") " pod="opendatahub/test-curl-7737-140142029905712" Sep 06 06:10:52.663956 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:52.663925 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glqwj\" (UniqueName: \"kubernetes.io/projected/ef8526b2-45c8-48f7-9347-77bcdb8cb957-kube-api-access-glqwj\") pod \"test-curl-7737-140142029905712\" (UID: \"ef8526b2-45c8-48f7-9347-77bcdb8cb957\") " pod="opendatahub/test-curl-7737-140142029905712" Sep 06 06:10:52.749013 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:52.748923 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-140142029905712" Sep 06 06:10:53.022959 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:53.022932 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7737-140142029905712"] Sep 06 06:10:53.027699 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:10:53.027605 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef8526b2_45c8_48f7_9347_77bcdb8cb957.slice/crio-0510d7c744ded1be49eaf04cc404f0383f7ba1e91acbe2f00b3c6b76fd651383 WatchSource:0}: Error finding container 0510d7c744ded1be49eaf04cc404f0383f7ba1e91acbe2f00b3c6b76fd651383: Status 404 returned error can't find the container with id 0510d7c744ded1be49eaf04cc404f0383f7ba1e91acbe2f00b3c6b76fd651383 Sep 06 06:10:53.518476 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:53.518441 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7737-140142029905712" event={"ID":"ef8526b2-45c8-48f7-9347-77bcdb8cb957","Type":"ContainerStarted","Data":"0510d7c744ded1be49eaf04cc404f0383f7ba1e91acbe2f00b3c6b76fd651383"} Sep 06 06:10:54.286397 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:54.286360 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:10:55.193267 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.193230 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn"] Sep 06 06:10:55.196357 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.196339 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.198496 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.198472 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-d5939f9b-kserve-self-signed-certs\"" Sep 06 06:10:55.204787 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.204765 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn"] Sep 06 06:10:55.283318 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.283293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-tmp-dir\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.283394 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.283325 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-home\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.283394 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.283344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-model-cache\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.283394 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.283363 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvlf9\" (UniqueName: \"kubernetes.io/projected/7397862c-8980-4407-a081-5d8e56326103-kube-api-access-jvlf9\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.283498 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.283483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-dshm\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.283562 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.283529 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7397862c-8980-4407-a081-5d8e56326103-tls-certs\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.384322 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.384280 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-tmp-dir\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.384494 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.384335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-home\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.384494 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.384357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-model-cache\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.384494 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.384378 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvlf9\" (UniqueName: \"kubernetes.io/projected/7397862c-8980-4407-a081-5d8e56326103-kube-api-access-jvlf9\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.384494 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.384436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-dshm\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.384494 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.384484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7397862c-8980-4407-a081-5d8e56326103-tls-certs\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.384736 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.384696 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-tmp-dir\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.384736 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.384728 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-home\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.384882 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.384862 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-model-cache\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.387317 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.387286 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-dshm\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.387432 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.387398 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7397862c-8980-4407-a081-5d8e56326103-tls-certs\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.396506 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.396481 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvlf9\" (UniqueName: \"kubernetes.io/projected/7397862c-8980-4407-a081-5d8e56326103-kube-api-access-jvlf9\") pod \"e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.506524 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.506444 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:10:55.530838 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.530808 2578 generic.go:356] "Generic (PLEG): container finished" podID="ef8526b2-45c8-48f7-9347-77bcdb8cb957" containerID="a8171ff819ce9cef0bf7eeff2f9dffc4212ca29dbedcd21dfcc766e9e859e6ac" exitCode=0 Sep 06 06:10:55.530999 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.530864 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7737-140142029905712" event={"ID":"ef8526b2-45c8-48f7-9347-77bcdb8cb957","Type":"ContainerDied","Data":"a8171ff819ce9cef0bf7eeff2f9dffc4212ca29dbedcd21dfcc766e9e859e6ac"} Sep 06 06:10:55.781865 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:55.781841 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn"] Sep 06 06:10:55.784241 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:10:55.784215 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7397862c_8980_4407_a081_5d8e56326103.slice/crio-fffc1d2ec816ab0e830b47ad28af0be1f75c3910ab2ba9895b0ffe5fe45a7774 WatchSource:0}: Error finding container fffc1d2ec816ab0e830b47ad28af0be1f75c3910ab2ba9895b0ffe5fe45a7774: Status 404 returned error can't find the container with id fffc1d2ec816ab0e830b47ad28af0be1f75c3910ab2ba9895b0ffe5fe45a7774 Sep 06 06:10:56.537521 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.537487 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" event={"ID":"7397862c-8980-4407-a081-5d8e56326103","Type":"ContainerStarted","Data":"aed6affe9678a6ee77f9f465c0f3f19a5d34b059fefa3e9fc2c4beac0bdc8d01"} Sep 06 06:10:56.538027 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.537539 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" event={"ID":"7397862c-8980-4407-a081-5d8e56326103","Type":"ContainerStarted","Data":"fffc1d2ec816ab0e830b47ad28af0be1f75c3910ab2ba9895b0ffe5fe45a7774"} Sep 06 06:10:56.553381 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.553333 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" podStartSLOduration=1.5533147409999999 podStartE2EDuration="1.553314741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:10:56.551508637 +0000 UTC m=+1021.949839129" watchObservedRunningTime="2026-09-06 06:10:56.553314741 +0000 UTC m=+1021.951645208" Sep 06 06:10:56.679951 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.679923 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-140142029905712" Sep 06 06:10:56.744451 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.744426 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7737-140142029905712_ef8526b2-45c8-48f7-9347-77bcdb8cb957/test-curl-7737-140142029905712/0.log" Sep 06 06:10:56.771792 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.771759 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7737-140142029905712"] Sep 06 06:10:56.785586 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.785558 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7737-140142029905712"] Sep 06 06:10:56.798154 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.798087 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glqwj\" (UniqueName: \"kubernetes.io/projected/ef8526b2-45c8-48f7-9347-77bcdb8cb957-kube-api-access-glqwj\") pod \"ef8526b2-45c8-48f7-9347-77bcdb8cb957\" (UID: \"ef8526b2-45c8-48f7-9347-77bcdb8cb957\") " Sep 06 06:10:56.800374 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.800347 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8526b2-45c8-48f7-9347-77bcdb8cb957-kube-api-access-glqwj" (OuterVolumeSpecName: "kube-api-access-glqwj") pod "ef8526b2-45c8-48f7-9347-77bcdb8cb957" (UID: "ef8526b2-45c8-48f7-9347-77bcdb8cb957"). InnerVolumeSpecName "kube-api-access-glqwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:10:56.898384 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.898350 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7737-140142029905712"] Sep 06 06:10:56.898725 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.898711 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8526b2-45c8-48f7-9347-77bcdb8cb957" containerName="test-curl-7737-140142029905712" Sep 06 06:10:56.898725 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.898725 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8526b2-45c8-48f7-9347-77bcdb8cb957" containerName="test-curl-7737-140142029905712" Sep 06 06:10:56.898817 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.898730 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glqwj\" (UniqueName: \"kubernetes.io/projected/ef8526b2-45c8-48f7-9347-77bcdb8cb957-kube-api-access-glqwj\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:10:56.898817 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.898794 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8526b2-45c8-48f7-9347-77bcdb8cb957" containerName="test-curl-7737-140142029905712" Sep 06 06:10:56.901739 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.901724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-140142029905712" Sep 06 06:10:56.908220 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:56.908192 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7737-140142029905712"] Sep 06 06:10:57.000164 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:57.000132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z4lf\" (UniqueName: \"kubernetes.io/projected/b48b4a22-c098-4c6a-b295-bfe29652399b-kube-api-access-7z4lf\") pod \"test-curl-7737-140142029905712\" (UID: \"b48b4a22-c098-4c6a-b295-bfe29652399b\") " pod="opendatahub/test-curl-7737-140142029905712" Sep 06 06:10:57.101476 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:57.101389 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4lf\" (UniqueName: \"kubernetes.io/projected/b48b4a22-c098-4c6a-b295-bfe29652399b-kube-api-access-7z4lf\") pod \"test-curl-7737-140142029905712\" (UID: \"b48b4a22-c098-4c6a-b295-bfe29652399b\") " pod="opendatahub/test-curl-7737-140142029905712" Sep 06 06:10:57.108691 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:57.108670 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z4lf\" (UniqueName: \"kubernetes.io/projected/b48b4a22-c098-4c6a-b295-bfe29652399b-kube-api-access-7z4lf\") pod \"test-curl-7737-140142029905712\" (UID: \"b48b4a22-c098-4c6a-b295-bfe29652399b\") " pod="opendatahub/test-curl-7737-140142029905712" Sep 06 06:10:57.145535 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:57.145490 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef8526b2-45c8-48f7-9347-77bcdb8cb957" path="/var/lib/kubelet/pods/ef8526b2-45c8-48f7-9347-77bcdb8cb957/volumes" Sep 06 06:10:57.213162 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:57.213126 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-140142029905712" Sep 06 06:10:57.484680 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:57.484655 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7737-140142029905712"] Sep 06 06:10:57.486437 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:10:57.486403 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb48b4a22_c098_4c6a_b295_bfe29652399b.slice/crio-ba94049dd208464187b09cbacd61dc46a9f8c9aba89b40bd0762715995d0ded5 WatchSource:0}: Error finding container ba94049dd208464187b09cbacd61dc46a9f8c9aba89b40bd0762715995d0ded5: Status 404 returned error can't find the container with id ba94049dd208464187b09cbacd61dc46a9f8c9aba89b40bd0762715995d0ded5 Sep 06 06:10:57.543592 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:57.543566 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-140142029905712" Sep 06 06:10:57.544507 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:57.543611 2578 scope.go:117] "RemoveContainer" containerID="a8171ff819ce9cef0bf7eeff2f9dffc4212ca29dbedcd21dfcc766e9e859e6ac" Sep 06 06:10:57.544964 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:57.544936 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7737-140142029905712" event={"ID":"b48b4a22-c098-4c6a-b295-bfe29652399b","Type":"ContainerStarted","Data":"ba94049dd208464187b09cbacd61dc46a9f8c9aba89b40bd0762715995d0ded5"} Sep 06 06:10:58.552381 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.552293 2578 generic.go:356] "Generic (PLEG): container finished" podID="b48b4a22-c098-4c6a-b295-bfe29652399b" containerID="e8f6cbadfec095425411d7d6c8f867904b7cf83f889c61cdd1f9aec0c0ff345b" exitCode=0 Sep 06 06:10:58.552823 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.552421 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7737-140142029905712" event={"ID":"b48b4a22-c098-4c6a-b295-bfe29652399b","Type":"ContainerDied","Data":"e8f6cbadfec095425411d7d6c8f867904b7cf83f889c61cdd1f9aec0c0ff345b"} Sep 06 06:10:58.748843 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.748814 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh"] Sep 06 06:10:58.752356 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.752334 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.754505 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.754487 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5da01692-openshift-default-dockercfg-qwbx4\"" Sep 06 06:10:58.765637 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.765612 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh"] Sep 06 06:10:58.818617 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.818532 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-workload-certs\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.818617 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.818572 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-credential-socket\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.818617 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.818595 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-envoy\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.818837 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.818683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-data\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.818837 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.818712 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbzdn\" (UniqueName: \"kubernetes.io/projected/b6256f6d-461c-403f-85f3-686fa2c0e599-kube-api-access-sbzdn\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.818837 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.818761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-token\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.818837 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.818807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-workload-socket\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.819332 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.818838 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6256f6d-461c-403f-85f3-686fa2c0e599-istiod-ca-cert\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.819332 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.818869 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-podinfo\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.919990 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.919941 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-workload-certs\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.920187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.920003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-credential-socket\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.920187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.920024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-envoy\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.920187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.920055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-data\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.920187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.920075 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbzdn\" (UniqueName: \"kubernetes.io/projected/b6256f6d-461c-403f-85f3-686fa2c0e599-kube-api-access-sbzdn\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.920187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.920104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-token\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.920187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.920161 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-workload-socket\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.920509 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.920206 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6256f6d-461c-403f-85f3-686fa2c0e599-istiod-ca-cert\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.920509 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.920251 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-podinfo\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.920509 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.920401 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-workload-certs\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.920509 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.920436 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-data\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.920704 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.920564 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-workload-socket\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.920704 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.920614 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-credential-socket\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.921000 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.920929 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6256f6d-461c-403f-85f3-686fa2c0e599-istiod-ca-cert\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.922410 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.922390 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-envoy\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.922803 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.922787 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-podinfo\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.927662 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.927638 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-token\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:58.927963 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:58.927945 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbzdn\" (UniqueName: \"kubernetes.io/projected/b6256f6d-461c-403f-85f3-686fa2c0e599-kube-api-access-sbzdn\") pod \"e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:59.062543 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:59.062509 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:10:59.338004 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:59.337972 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh"] Sep 06 06:10:59.339973 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:10:59.339946 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6256f6d_461c_403f_85f3_686fa2c0e599.slice/crio-471491f6e547221d5f74eddebb0e5f4ae71f8b89cea0f69d3f97e49be2de72e5 WatchSource:0}: Error finding container 471491f6e547221d5f74eddebb0e5f4ae71f8b89cea0f69d3f97e49be2de72e5: Status 404 returned error can't find the container with id 471491f6e547221d5f74eddebb0e5f4ae71f8b89cea0f69d3f97e49be2de72e5 Sep 06 06:10:59.342714 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:59.342683 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:10:59.342797 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:59.342760 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:10:59.342863 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:59.342803 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:10:59.561720 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:59.561680 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" event={"ID":"b6256f6d-461c-403f-85f3-686fa2c0e599","Type":"ContainerStarted","Data":"d6019e7b7e9a44895662e4993fe2e56e87392db7feba06105561e9693caca615"} Sep 06 06:10:59.562228 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:59.561747 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" event={"ID":"b6256f6d-461c-403f-85f3-686fa2c0e599","Type":"ContainerStarted","Data":"471491f6e547221d5f74eddebb0e5f4ae71f8b89cea0f69d3f97e49be2de72e5"} Sep 06 06:10:59.581377 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:59.581323 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" podStartSLOduration=1.5813046229999999 podStartE2EDuration="1.581304623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:10:59.579019994 +0000 UTC m=+1024.977350479" watchObservedRunningTime="2026-09-06 06:10:59.581304623 +0000 UTC m=+1024.979635090" Sep 06 06:10:59.714790 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:59.714765 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-140142029905712" Sep 06 06:10:59.731449 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:59.731420 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7737-140142029905712_b48b4a22-c098-4c6a-b295-bfe29652399b/test-curl-7737-140142029905712/0.log" Sep 06 06:10:59.750017 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:59.749991 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7737-140142029905712"] Sep 06 06:10:59.753196 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:59.753169 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7737-140142029905712"] Sep 06 06:10:59.828872 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:59.828835 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7z4lf\" (UniqueName: \"kubernetes.io/projected/b48b4a22-c098-4c6a-b295-bfe29652399b-kube-api-access-7z4lf\") pod \"b48b4a22-c098-4c6a-b295-bfe29652399b\" (UID: \"b48b4a22-c098-4c6a-b295-bfe29652399b\") " Sep 06 06:10:59.831126 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:59.831106 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48b4a22-c098-4c6a-b295-bfe29652399b-kube-api-access-7z4lf" (OuterVolumeSpecName: "kube-api-access-7z4lf") pod "b48b4a22-c098-4c6a-b295-bfe29652399b" (UID: "b48b4a22-c098-4c6a-b295-bfe29652399b"). InnerVolumeSpecName "kube-api-access-7z4lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:10:59.930512 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:10:59.930430 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z4lf\" (UniqueName: \"kubernetes.io/projected/b48b4a22-c098-4c6a-b295-bfe29652399b-kube-api-access-7z4lf\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:11:00.063699 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.063664 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:11:00.068662 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.068636 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:11:00.516955 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.513659 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs"] Sep 06 06:11:00.516955 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.514580 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b48b4a22-c098-4c6a-b295-bfe29652399b" containerName="test-curl-7737-140142029905712" Sep 06 06:11:00.516955 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.514601 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b48b4a22-c098-4c6a-b295-bfe29652399b" containerName="test-curl-7737-140142029905712" Sep 06 06:11:00.516955 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.514830 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b48b4a22-c098-4c6a-b295-bfe29652399b" containerName="test-curl-7737-140142029905712" Sep 06 06:11:00.518924 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.518892 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.522319 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.522061 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-e75768-openshift-default-dockercfg-x9x2g\"" Sep 06 06:11:00.524667 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.524627 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs"] Sep 06 06:11:00.567430 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.567353 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba94049dd208464187b09cbacd61dc46a9f8c9aba89b40bd0762715995d0ded5" Sep 06 06:11:00.567430 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.567358 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-140142029905712" Sep 06 06:11:00.567826 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.567803 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:11:00.568740 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.568721 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:11:00.585878 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:11:00.585851 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7737-140142029905712\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="b48b4a22-c098-4c6a-b295-bfe29652399b" pod="opendatahub/test-curl-7737-140142029905712" Sep 06 06:11:00.639090 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.639039 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-envoy\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.639090 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.639094 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-data\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.639285 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.639111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-podinfo\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.639285 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.639245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-workload-certs\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.639383 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.639282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p89j8\" (UniqueName: \"kubernetes.io/projected/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-kube-api-access-p89j8\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.639383 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.639369 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istiod-ca-cert\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.639462 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.639398 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-token\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.639534 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.639514 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-credential-socket\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.639589 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.639548 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-workload-socket\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.740540 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.740504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-token\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.740739 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.740565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-credential-socket\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.740739 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.740586 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-workload-socket\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.740739 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.740619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-envoy\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.740739 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.740654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-data\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.740967 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.740797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-podinfo\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.740967 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.740858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-workload-certs\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.740967 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.740874 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p89j8\" (UniqueName: \"kubernetes.io/projected/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-kube-api-access-p89j8\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.740967 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.740951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istiod-ca-cert\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.741171 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.741036 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-credential-socket\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.741171 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.741099 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-workload-socket\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.741171 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.741109 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-data\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.741395 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.741329 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-workload-certs\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.741638 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.741617 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istiod-ca-cert\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.743201 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.743181 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-envoy\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.743599 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.743580 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-podinfo\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.748102 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.748074 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-token\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.748231 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.748212 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p89j8\" (UniqueName: \"kubernetes.io/projected/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-kube-api-access-p89j8\") pod \"e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:00.832929 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:00.832827 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:01.123626 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:01.123549 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs"] Sep 06 06:11:01.127595 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:11:01.127569 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97058a65_bc0c_44d0_86d7_1320b9d8a5e8.slice/crio-c797ee7bb811d49eeb4e7aa57130ed3afa028513a62a44f522287e3eeeed8035 WatchSource:0}: Error finding container c797ee7bb811d49eeb4e7aa57130ed3afa028513a62a44f522287e3eeeed8035: Status 404 returned error can't find the container with id c797ee7bb811d49eeb4e7aa57130ed3afa028513a62a44f522287e3eeeed8035 Sep 06 06:11:01.129982 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:01.129894 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:01.130069 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:01.130020 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:01.130069 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:01.130050 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:01.148635 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:01.148610 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b48b4a22-c098-4c6a-b295-bfe29652399b" path="/var/lib/kubelet/pods/b48b4a22-c098-4c6a-b295-bfe29652399b/volumes" Sep 06 06:11:01.576593 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:01.576551 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" event={"ID":"97058a65-bc0c-44d0-86d7-1320b9d8a5e8","Type":"ContainerStarted","Data":"8d395ad83ce99da7e5af6d1e79e7e4354e0a833eceeacef77e0b1a986c0a6338"} Sep 06 06:11:01.577076 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:01.576605 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" event={"ID":"97058a65-bc0c-44d0-86d7-1320b9d8a5e8","Type":"ContainerStarted","Data":"c797ee7bb811d49eeb4e7aa57130ed3afa028513a62a44f522287e3eeeed8035"} Sep 06 06:11:01.599546 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:01.599494 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" podStartSLOduration=1.599475623 podStartE2EDuration="1.599475623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:11:01.597201477 +0000 UTC m=+1026.995531968" watchObservedRunningTime="2026-09-06 06:11:01.599475623 +0000 UTC m=+1026.997806089" Sep 06 06:11:01.833535 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:01.833447 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:02.838053 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:02.838015 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:03.589588 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:03.589557 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:03.590638 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:03.590614 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:05.507514 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:05.507478 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:11:05.508262 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:05.508244 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:11:05.523244 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:05.523215 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:11:05.611922 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:05.611879 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:11:09.589796 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:09.589706 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:11:13.644180 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:13.644152 2578 generic.go:356] "Generic (PLEG): container finished" podID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerID="f86d4da5d4968e152942f0e3f0b37e044c235b256d31bb13d8a3d5312279afb9" exitCode=0 Sep 06 06:11:13.644664 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:13.644220 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" event={"ID":"cd5046cd-6f68-45bf-8e6b-59444f9af36d","Type":"ContainerDied","Data":"f86d4da5d4968e152942f0e3f0b37e044c235b256d31bb13d8a3d5312279afb9"} Sep 06 06:11:13.645070 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:13.645055 2578 scope.go:117] "RemoveContainer" containerID="f86d4da5d4968e152942f0e3f0b37e044c235b256d31bb13d8a3d5312279afb9" Sep 06 06:11:13.646139 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:13.646112 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:13.646187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:13.646179 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:13.646222 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:13.646210 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:14.654950 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:14.653896 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" event={"ID":"cd5046cd-6f68-45bf-8e6b-59444f9af36d","Type":"ContainerStarted","Data":"3805697d131e86a9ea7fd3332aa3b1585c9f5929b5be3a1d0da92a9c4c035263"} Sep 06 06:11:15.124600 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:15.124570 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:11:15.124894 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:15.124877 2578 patch_prober.go:28] interesting pod/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 06 06:11:15.125051 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:15.125033 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" podUID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 06 06:11:16.124873 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:16.124421 2578 patch_prober.go:28] interesting pod/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 06 06:11:16.124873 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:16.124480 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" podUID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 06 06:11:17.124114 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:17.124071 2578 patch_prober.go:28] interesting pod/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 06 06:11:17.124308 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:17.124134 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" podUID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 06 06:11:18.124675 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:18.124574 2578 patch_prober.go:28] interesting pod/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 06 06:11:18.124675 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:18.124652 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" podUID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 06 06:11:18.910597 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:18.910550 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp"] Sep 06 06:11:18.918330 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:18.918306 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:18.920817 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:18.920625 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0bce0b-openshift-default-dockercfg-hq4fp\"" Sep 06 06:11:18.929192 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:18.927981 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp"] Sep 06 06:11:18.945376 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:18.945311 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-workload-socket\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:18.945376 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:18.945372 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-token\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:18.945610 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:18.945406 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-podinfo\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:18.945610 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:18.945484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-data\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:18.945610 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:18.945534 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452e293c-1514-4cca-b4c8-d0d4e10a0290-istiod-ca-cert\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:18.945610 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:18.945566 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-credential-socket\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:18.945610 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:18.945586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-workload-certs\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:18.945883 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:18.945621 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-246zx\" (UniqueName: \"kubernetes.io/projected/452e293c-1514-4cca-b4c8-d0d4e10a0290-kube-api-access-246zx\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:18.945883 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:18.945694 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-envoy\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.047146 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.047114 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-envoy\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.047485 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.047466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-workload-socket\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.048336 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.047995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-token\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.048511 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.048492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-podinfo\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.048729 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.047931 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-workload-socket\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.048894 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.048863 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-data\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.049424 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.049285 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-data\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.050529 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.050281 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452e293c-1514-4cca-b4c8-d0d4e10a0290-istiod-ca-cert\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.050529 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.050492 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-envoy\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.050731 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.050710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452e293c-1514-4cca-b4c8-d0d4e10a0290-istiod-ca-cert\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.050879 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.050861 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-credential-socket\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.051342 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.051314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-workload-certs\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.051860 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.051249 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-credential-socket\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.051860 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.051724 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-workload-certs\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.052136 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.052116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-246zx\" (UniqueName: \"kubernetes.io/projected/452e293c-1514-4cca-b4c8-d0d4e10a0290-kube-api-access-246zx\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.059112 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.059045 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-token\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.062030 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.062004 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-246zx\" (UniqueName: \"kubernetes.io/projected/452e293c-1514-4cca-b4c8-d0d4e10a0290-kube-api-access-246zx\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.065360 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.065315 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-podinfo\") pod \"e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.124736 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.124492 2578 patch_prober.go:28] interesting pod/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 06 06:11:19.124736 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.124558 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" podUID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 06 06:11:19.236226 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.236131 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:19.557747 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.557652 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp"] Sep 06 06:11:19.580349 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.580298 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:19.580485 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.580397 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:19.580485 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.580447 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:19.634194 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.634160 2578 patch_prober.go:28] interesting pod/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 06 06:11:19.634334 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.634227 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" podUID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 06 06:11:19.699413 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.699379 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" event={"ID":"452e293c-1514-4cca-b4c8-d0d4e10a0290","Type":"ContainerStarted","Data":"a86d10d4e6d40648ffa93bcff3fe0942f0d47d11eb84d948a5944985d5d4dbbd"} Sep 06 06:11:19.699580 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.699432 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" event={"ID":"452e293c-1514-4cca-b4c8-d0d4e10a0290","Type":"ContainerStarted","Data":"3e3f541a3ce17d92d2db3680b5188bbeadad3c24d22d80f621f3b3499ea241db"} Sep 06 06:11:19.718100 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:19.718048 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" podStartSLOduration=1.718031026 podStartE2EDuration="1.718031026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:11:19.71653523 +0000 UTC m=+1045.114865718" watchObservedRunningTime="2026-09-06 06:11:19.718031026 +0000 UTC m=+1045.116361494" Sep 06 06:11:20.124425 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:20.124385 2578 patch_prober.go:28] interesting pod/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Sep 06 06:11:20.124638 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:20.124454 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" podUID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Sep 06 06:11:20.237241 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:20.237197 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:20.244143 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:20.244116 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:20.704359 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:20.704324 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:20.705455 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:20.705434 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:21.128970 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.128938 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:11:21.129396 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.129374 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:11:21.130239 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.130218 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:11:21.711022 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.710947 2578 generic.go:356] "Generic (PLEG): container finished" podID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerID="8d395ad83ce99da7e5af6d1e79e7e4354e0a833eceeacef77e0b1a986c0a6338" exitCode=0 Sep 06 06:11:21.711502 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.711026 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" event={"ID":"97058a65-bc0c-44d0-86d7-1320b9d8a5e8","Type":"ContainerDied","Data":"8d395ad83ce99da7e5af6d1e79e7e4354e0a833eceeacef77e0b1a986c0a6338"} Sep 06 06:11:21.711570 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.711537 2578 scope.go:117] "RemoveContainer" containerID="8d395ad83ce99da7e5af6d1e79e7e4354e0a833eceeacef77e0b1a986c0a6338" Sep 06 06:11:21.715177 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.713210 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:21.715177 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.713350 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerID="d6019e7b7e9a44895662e4993fe2e56e87392db7feba06105561e9693caca615" exitCode=0 Sep 06 06:11:21.715177 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.713402 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:21.715177 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.713504 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:21.715177 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.714233 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" event={"ID":"b6256f6d-461c-403f-85f3-686fa2c0e599","Type":"ContainerDied","Data":"d6019e7b7e9a44895662e4993fe2e56e87392db7feba06105561e9693caca615"} Sep 06 06:11:21.721436 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.721410 2578 scope.go:117] "RemoveContainer" containerID="d6019e7b7e9a44895662e4993fe2e56e87392db7feba06105561e9693caca615" Sep 06 06:11:21.724493 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.724468 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:21.724606 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.724538 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:21.724606 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.724580 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:21.881024 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:21.880983 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:11:22.720413 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:22.720190 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" event={"ID":"97058a65-bc0c-44d0-86d7-1320b9d8a5e8","Type":"ContainerStarted","Data":"e16680b378cbb785588bf47b4bb679bed88d7889a220ec752e5f16c332c47172"} Sep 06 06:11:22.721846 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:22.721818 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" event={"ID":"b6256f6d-461c-403f-85f3-686fa2c0e599","Type":"ContainerStarted","Data":"793416d39bcde88a2ddc1cc35db6d3748bb55863b370e3b0a88e4415163abb71"} Sep 06 06:11:22.833486 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:22.833449 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:22.833850 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:22.833825 2578 patch_prober.go:28] interesting pod/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 06 06:11:22.834070 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:22.834043 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" podUID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 06 06:11:23.063268 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:23.063212 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:11:23.063512 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:23.063249 2578 patch_prober.go:28] interesting pod/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 06 06:11:23.063680 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:23.063654 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" podUID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 06 06:11:23.834074 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:23.834038 2578 patch_prober.go:28] interesting pod/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 06 06:11:23.834478 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:23.834092 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" podUID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 06 06:11:24.067929 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:24.063924 2578 patch_prober.go:28] interesting pod/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 06 06:11:24.067929 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:24.064032 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" podUID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 06 06:11:24.833601 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:24.833562 2578 patch_prober.go:28] interesting pod/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 06 06:11:24.833825 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:24.833628 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" podUID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 06 06:11:25.063329 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:25.063283 2578 patch_prober.go:28] interesting pod/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 06 06:11:25.063807 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:25.063354 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" podUID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 06 06:11:25.833886 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:25.833853 2578 patch_prober.go:28] interesting pod/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 06 06:11:25.834110 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:25.833940 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" podUID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 06 06:11:26.063347 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:26.063299 2578 patch_prober.go:28] interesting pod/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 06 06:11:26.063843 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:26.063371 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" podUID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 06 06:11:26.336358 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:26.336328 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs"] Sep 06 06:11:26.342164 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:26.342146 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs" Sep 06 06:11:26.344380 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:26.344363 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:11:26.344509 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:26.344363 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:11:26.346168 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:26.346146 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs"] Sep 06 06:11:26.479931 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:26.477529 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/274088bd-9612-42b4-8f8d-de95c92a31ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5da01692-tvsrs\" (UID: \"274088bd-9612-42b4-8f8d-de95c92a31ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs" Sep 06 06:11:26.578857 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:26.578800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/274088bd-9612-42b4-8f8d-de95c92a31ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5da01692-tvsrs\" (UID: \"274088bd-9612-42b4-8f8d-de95c92a31ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs" Sep 06 06:11:26.579811 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:26.579779 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/274088bd-9612-42b4-8f8d-de95c92a31ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5da01692-tvsrs\" (UID: \"274088bd-9612-42b4-8f8d-de95c92a31ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs" Sep 06 06:11:26.678999 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:26.678881 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs" Sep 06 06:11:26.833310 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:26.833278 2578 patch_prober.go:28] interesting pod/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 06 06:11:26.833443 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:26.833338 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" podUID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 06 06:11:27.023696 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:27.023671 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs"] Sep 06 06:11:27.025275 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:11:27.025247 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274088bd_9612_42b4_8f8d_de95c92a31ff.slice/crio-916bc12eddadd5ae5c1f9e0664a2fb1e3d4c513b01a300264afdf1c62073185d WatchSource:0}: Error finding container 916bc12eddadd5ae5c1f9e0664a2fb1e3d4c513b01a300264afdf1c62073185d: Status 404 returned error can't find the container with id 916bc12eddadd5ae5c1f9e0664a2fb1e3d4c513b01a300264afdf1c62073185d Sep 06 06:11:27.063187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:27.063148 2578 patch_prober.go:28] interesting pod/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 06 06:11:27.063383 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:27.063217 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" podUID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 06 06:11:27.753312 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:27.753253 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs" event={"ID":"274088bd-9612-42b4-8f8d-de95c92a31ff","Type":"ContainerStarted","Data":"916bc12eddadd5ae5c1f9e0664a2fb1e3d4c513b01a300264afdf1c62073185d"} Sep 06 06:11:27.834088 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:27.834048 2578 patch_prober.go:28] interesting pod/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Sep 06 06:11:27.834256 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:27.834122 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" podUID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Sep 06 06:11:28.063541 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:28.063500 2578 patch_prober.go:28] interesting pod/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Sep 06 06:11:28.064017 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:28.063563 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" podUID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Sep 06 06:11:28.837796 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:28.837763 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:28.838210 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:28.838186 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:28.839086 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:28.839049 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:11:29.215148 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:29.214877 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:11:29.215579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:29.215372 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:11:29.216426 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:29.216407 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:11:29.770197 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:29.770156 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs" event={"ID":"274088bd-9612-42b4-8f8d-de95c92a31ff","Type":"ContainerStarted","Data":"f704e5517983739d17c198f48c8b012ffd493d8a529c959684a4ae0195d51cc1"} Sep 06 06:11:29.786315 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:29.786262 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs" podStartSLOduration=1.456971854 podStartE2EDuration="3.786243487s" totalImagesPullingTime="2.329271633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:11:29.783847774 +0000 UTC m=+1055.182178259" watchObservedRunningTime="2026-09-06 06:11:29.786243487 +0000 UTC m=+1055.184573954" Sep 06 06:11:30.783445 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:30.783412 2578 generic.go:356] "Generic (PLEG): container finished" podID="274088bd-9612-42b4-8f8d-de95c92a31ff" containerID="f704e5517983739d17c198f48c8b012ffd493d8a529c959684a4ae0195d51cc1" exitCode=0 Sep 06 06:11:30.783818 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:30.783466 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs" event={"ID":"274088bd-9612-42b4-8f8d-de95c92a31ff","Type":"ContainerDied","Data":"f704e5517983739d17c198f48c8b012ffd493d8a529c959684a4ae0195d51cc1"} Sep 06 06:11:31.941561 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:31.941532 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs" Sep 06 06:11:32.078565 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:32.078463 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/274088bd-9612-42b4-8f8d-de95c92a31ff-maas-api-service-ca\") pod \"274088bd-9612-42b4-8f8d-de95c92a31ff\" (UID: \"274088bd-9612-42b4-8f8d-de95c92a31ff\") " Sep 06 06:11:32.078817 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:32.078790 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/274088bd-9612-42b4-8f8d-de95c92a31ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "274088bd-9612-42b4-8f8d-de95c92a31ff" (UID: "274088bd-9612-42b4-8f8d-de95c92a31ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:11:32.180072 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:32.180039 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/274088bd-9612-42b4-8f8d-de95c92a31ff-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:11:32.796612 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:32.796580 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs" Sep 06 06:11:32.796781 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:32.796609 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs" event={"ID":"274088bd-9612-42b4-8f8d-de95c92a31ff","Type":"ContainerDied","Data":"916bc12eddadd5ae5c1f9e0664a2fb1e3d4c513b01a300264afdf1c62073185d"} Sep 06 06:11:32.796781 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:32.796659 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="916bc12eddadd5ae5c1f9e0664a2fb1e3d4c513b01a300264afdf1c62073185d" Sep 06 06:11:35.976813 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:35.976775 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:11:36.746099 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:36.746061 2578 patch_prober.go:28] interesting pod/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 06 06:11:36.746350 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:36.746125 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" podUID="452e293c-1514-4cca-b4c8-d0d4e10a0290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 06 06:11:38.829301 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:38.829265 2578 generic.go:356] "Generic (PLEG): container finished" podID="452e293c-1514-4cca-b4c8-d0d4e10a0290" containerID="a86d10d4e6d40648ffa93bcff3fe0942f0d47d11eb84d948a5944985d5d4dbbd" exitCode=0 Sep 06 06:11:38.829709 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:38.829342 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" event={"ID":"452e293c-1514-4cca-b4c8-d0d4e10a0290","Type":"ContainerDied","Data":"a86d10d4e6d40648ffa93bcff3fe0942f0d47d11eb84d948a5944985d5d4dbbd"} Sep 06 06:11:38.829755 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:38.829745 2578 scope.go:117] "RemoveContainer" containerID="a86d10d4e6d40648ffa93bcff3fe0942f0d47d11eb84d948a5944985d5d4dbbd" Sep 06 06:11:38.830998 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:38.830961 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:38.831108 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:38.831039 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:38.831108 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:38.831086 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:39.837131 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:39.837103 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" event={"ID":"452e293c-1514-4cca-b4c8-d0d4e10a0290","Type":"ContainerStarted","Data":"861a32240a772cef8c790e3ba0ea9d434907ff45c38d3ea1a55f189ee3551692"} Sep 06 06:11:40.236429 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:40.236336 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:40.236604 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:40.236543 2578 patch_prober.go:28] interesting pod/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 06 06:11:40.236676 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:40.236598 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" podUID="452e293c-1514-4cca-b4c8-d0d4e10a0290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 06 06:11:41.237227 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:41.237191 2578 patch_prober.go:28] interesting pod/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 06 06:11:41.237641 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:41.237247 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" podUID="452e293c-1514-4cca-b4c8-d0d4e10a0290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 06 06:11:42.237256 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:42.237218 2578 patch_prober.go:28] interesting pod/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 06 06:11:42.237632 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:42.237272 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" podUID="452e293c-1514-4cca-b4c8-d0d4e10a0290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 06 06:11:43.066561 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:43.066526 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs"] Sep 06 06:11:43.077237 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:43.077199 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5da01692-tvsrs"] Sep 06 06:11:43.144916 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:43.144875 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="274088bd-9612-42b4-8f8d-de95c92a31ff" path="/var/lib/kubelet/pods/274088bd-9612-42b4-8f8d-de95c92a31ff/volumes" Sep 06 06:11:43.187824 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:43.187785 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh"] Sep 06 06:11:43.188119 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:43.188090 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" podUID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerName="istio-proxy" containerID="cri-o://793416d39bcde88a2ddc1cc35db6d3748bb55863b370e3b0a88e4415163abb71" gracePeriod=30 Sep 06 06:11:43.237135 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:43.237101 2578 patch_prober.go:28] interesting pod/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 06 06:11:43.237306 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:43.237159 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" podUID="452e293c-1514-4cca-b4c8-d0d4e10a0290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 06 06:11:44.118571 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.118541 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg"] Sep 06 06:11:44.119027 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.119013 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="274088bd-9612-42b4-8f8d-de95c92a31ff" containerName="revoke-keys" Sep 06 06:11:44.119129 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.119030 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="274088bd-9612-42b4-8f8d-de95c92a31ff" containerName="revoke-keys" Sep 06 06:11:44.119129 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.119103 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="274088bd-9612-42b4-8f8d-de95c92a31ff" containerName="revoke-keys" Sep 06 06:11:44.122434 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.122418 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.124584 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.124553 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-06e64f94-gw-openshift-default-dockercfg-htgqx\"" Sep 06 06:11:44.133458 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.133432 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg"] Sep 06 06:11:44.203426 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.203390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-workload-certs\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.203426 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.203429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-envoy\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.203665 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.203472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-workload-socket\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.203665 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.203625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-data\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.203746 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.203680 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istiod-ca-cert\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.203746 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.203735 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-podinfo\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.203843 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.203822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-credential-socket\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.203953 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.203859 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-token\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.204000 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.203949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv84g\" (UniqueName: \"kubernetes.io/projected/faee71d7-c3f8-4d18-88d4-e18b492a81a0-kube-api-access-nv84g\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.218598 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.218564 2578 patch_prober.go:28] interesting pod/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 06 06:11:44.218856 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.218830 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" podUID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 06 06:11:44.236963 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.236928 2578 patch_prober.go:28] interesting pod/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Sep 06 06:11:44.237159 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.237002 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" podUID="452e293c-1514-4cca-b4c8-d0d4e10a0290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Sep 06 06:11:44.304782 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.304744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv84g\" (UniqueName: \"kubernetes.io/projected/faee71d7-c3f8-4d18-88d4-e18b492a81a0-kube-api-access-nv84g\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.305277 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.304806 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-workload-certs\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.305277 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.304826 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-envoy\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.305277 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.304856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-workload-socket\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.305277 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.304885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-data\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.305277 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.304925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istiod-ca-cert\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.305277 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.304973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-podinfo\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.305277 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.305013 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-credential-socket\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.305277 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.305043 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-token\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.305673 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.305383 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-workload-certs\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.305673 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.305405 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-workload-socket\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.305673 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.305593 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-credential-socket\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.305673 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.305609 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-data\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.306276 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.306251 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istiod-ca-cert\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.307881 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.307847 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-envoy\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.307994 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.307961 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-podinfo\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.312216 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.312187 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-token\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.312423 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.312409 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv84g\" (UniqueName: \"kubernetes.io/projected/faee71d7-c3f8-4d18-88d4-e18b492a81a0-kube-api-access-nv84g\") pod \"e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.434675 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.434585 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:44.735968 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.735932 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg"] Sep 06 06:11:44.737336 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:11:44.737308 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaee71d7_c3f8_4d18_88d4_e18b492a81a0.slice/crio-43cdd75c53f56391520ecb79e4555046d1c988abf1c23ec02f7f62884f760fc7 WatchSource:0}: Error finding container 43cdd75c53f56391520ecb79e4555046d1c988abf1c23ec02f7f62884f760fc7: Status 404 returned error can't find the container with id 43cdd75c53f56391520ecb79e4555046d1c988abf1c23ec02f7f62884f760fc7 Sep 06 06:11:44.739565 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.739537 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:44.739650 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.739597 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:44.739650 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.739626 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:11:44.873872 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.873834 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" event={"ID":"faee71d7-c3f8-4d18-88d4-e18b492a81a0","Type":"ContainerStarted","Data":"b95b09d4ea317b245189fc3cbee53a744027679640fdbba74b9c0e92fca3a4a8"} Sep 06 06:11:44.874105 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.873944 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" event={"ID":"faee71d7-c3f8-4d18-88d4-e18b492a81a0","Type":"ContainerStarted","Data":"43cdd75c53f56391520ecb79e4555046d1c988abf1c23ec02f7f62884f760fc7"} Sep 06 06:11:44.893728 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:44.893665 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" podStartSLOduration=0.893643876 podStartE2EDuration="893.643876ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:11:44.890500104 +0000 UTC m=+1070.288830585" watchObservedRunningTime="2026-09-06 06:11:44.893643876 +0000 UTC m=+1070.291974343" Sep 06 06:11:45.266708 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:45.266672 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:45.267133 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:45.267112 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:45.268080 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:45.268059 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:11:45.435062 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:45.435031 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:45.440815 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:45.440788 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:45.879949 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:45.879894 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:45.880951 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:45.880932 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:11:48.451460 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.451401 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:11:48.548206 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.548175 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-workload-certs\") pod \"b6256f6d-461c-403f-85f3-686fa2c0e599\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " Sep 06 06:11:48.548206 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.548212 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbzdn\" (UniqueName: \"kubernetes.io/projected/b6256f6d-461c-403f-85f3-686fa2c0e599-kube-api-access-sbzdn\") pod \"b6256f6d-461c-403f-85f3-686fa2c0e599\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " Sep 06 06:11:48.548419 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.548280 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-podinfo\") pod \"b6256f6d-461c-403f-85f3-686fa2c0e599\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " Sep 06 06:11:48.548479 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.548429 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6256f6d-461c-403f-85f3-686fa2c0e599-istiod-ca-cert\") pod \"b6256f6d-461c-403f-85f3-686fa2c0e599\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " Sep 06 06:11:48.548479 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.548458 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b6256f6d-461c-403f-85f3-686fa2c0e599" (UID: "b6256f6d-461c-403f-85f3-686fa2c0e599"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:11:48.548583 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.548519 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-credential-socket\") pod \"b6256f6d-461c-403f-85f3-686fa2c0e599\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " Sep 06 06:11:48.548583 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.548567 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-workload-socket\") pod \"b6256f6d-461c-403f-85f3-686fa2c0e599\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " Sep 06 06:11:48.548682 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.548594 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-envoy\") pod \"b6256f6d-461c-403f-85f3-686fa2c0e599\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " Sep 06 06:11:48.548682 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.548620 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-token\") pod \"b6256f6d-461c-403f-85f3-686fa2c0e599\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " Sep 06 06:11:48.548682 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.548659 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-data\") pod \"b6256f6d-461c-403f-85f3-686fa2c0e599\" (UID: \"b6256f6d-461c-403f-85f3-686fa2c0e599\") " Sep 06 06:11:48.548829 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.548805 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b6256f6d-461c-403f-85f3-686fa2c0e599" (UID: "b6256f6d-461c-403f-85f3-686fa2c0e599"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:11:48.548887 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.548838 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6256f6d-461c-403f-85f3-686fa2c0e599-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b6256f6d-461c-403f-85f3-686fa2c0e599" (UID: "b6256f6d-461c-403f-85f3-686fa2c0e599"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:11:48.548887 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.548843 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b6256f6d-461c-403f-85f3-686fa2c0e599" (UID: "b6256f6d-461c-403f-85f3-686fa2c0e599"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:11:48.549035 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.549003 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-data" (OuterVolumeSpecName: "istio-data") pod "b6256f6d-461c-403f-85f3-686fa2c0e599" (UID: "b6256f6d-461c-403f-85f3-686fa2c0e599"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:11:48.549411 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.549330 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:11:48.549411 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.549353 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:11:48.549411 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.549368 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:11:48.549411 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.549380 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6256f6d-461c-403f-85f3-686fa2c0e599-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:11:48.549411 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.549393 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:11:48.550945 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.550897 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6256f6d-461c-403f-85f3-686fa2c0e599-kube-api-access-sbzdn" (OuterVolumeSpecName: "kube-api-access-sbzdn") pod "b6256f6d-461c-403f-85f3-686fa2c0e599" (UID: "b6256f6d-461c-403f-85f3-686fa2c0e599"). InnerVolumeSpecName "kube-api-access-sbzdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:11:48.551067 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.551017 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b6256f6d-461c-403f-85f3-686fa2c0e599" (UID: "b6256f6d-461c-403f-85f3-686fa2c0e599"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:11:48.551368 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.551337 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-token" (OuterVolumeSpecName: "istio-token") pod "b6256f6d-461c-403f-85f3-686fa2c0e599" (UID: "b6256f6d-461c-403f-85f3-686fa2c0e599"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:11:48.551430 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.551408 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b6256f6d-461c-403f-85f3-686fa2c0e599" (UID: "b6256f6d-461c-403f-85f3-686fa2c0e599"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:11:48.650620 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.650581 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbzdn\" (UniqueName: \"kubernetes.io/projected/b6256f6d-461c-403f-85f3-686fa2c0e599-kube-api-access-sbzdn\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:11:48.650620 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.650622 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:11:48.650810 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.650636 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:11:48.650810 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.650648 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6256f6d-461c-403f-85f3-686fa2c0e599-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:11:48.899338 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.899302 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerID="793416d39bcde88a2ddc1cc35db6d3748bb55863b370e3b0a88e4415163abb71" exitCode=0 Sep 06 06:11:48.899520 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.899360 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" Sep 06 06:11:48.899520 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.899390 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" event={"ID":"b6256f6d-461c-403f-85f3-686fa2c0e599","Type":"ContainerDied","Data":"793416d39bcde88a2ddc1cc35db6d3748bb55863b370e3b0a88e4415163abb71"} Sep 06 06:11:48.899520 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.899443 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh" event={"ID":"b6256f6d-461c-403f-85f3-686fa2c0e599","Type":"ContainerDied","Data":"471491f6e547221d5f74eddebb0e5f4ae71f8b89cea0f69d3f97e49be2de72e5"} Sep 06 06:11:48.899520 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.899468 2578 scope.go:117] "RemoveContainer" containerID="793416d39bcde88a2ddc1cc35db6d3748bb55863b370e3b0a88e4415163abb71" Sep 06 06:11:48.909534 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.909515 2578 scope.go:117] "RemoveContainer" containerID="d6019e7b7e9a44895662e4993fe2e56e87392db7feba06105561e9693caca615" Sep 06 06:11:48.917888 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.917869 2578 scope.go:117] "RemoveContainer" containerID="793416d39bcde88a2ddc1cc35db6d3748bb55863b370e3b0a88e4415163abb71" Sep 06 06:11:48.918258 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:11:48.918229 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"793416d39bcde88a2ddc1cc35db6d3748bb55863b370e3b0a88e4415163abb71\": container with ID starting with 793416d39bcde88a2ddc1cc35db6d3748bb55863b370e3b0a88e4415163abb71 not found: ID does not exist" containerID="793416d39bcde88a2ddc1cc35db6d3748bb55863b370e3b0a88e4415163abb71" Sep 06 06:11:48.918331 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.918271 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"793416d39bcde88a2ddc1cc35db6d3748bb55863b370e3b0a88e4415163abb71"} err="failed to get container status \"793416d39bcde88a2ddc1cc35db6d3748bb55863b370e3b0a88e4415163abb71\": rpc error: code = NotFound desc = could not find container \"793416d39bcde88a2ddc1cc35db6d3748bb55863b370e3b0a88e4415163abb71\": container with ID starting with 793416d39bcde88a2ddc1cc35db6d3748bb55863b370e3b0a88e4415163abb71 not found: ID does not exist" Sep 06 06:11:48.918331 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.918292 2578 scope.go:117] "RemoveContainer" containerID="d6019e7b7e9a44895662e4993fe2e56e87392db7feba06105561e9693caca615" Sep 06 06:11:48.918588 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:11:48.918569 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6019e7b7e9a44895662e4993fe2e56e87392db7feba06105561e9693caca615\": container with ID starting with d6019e7b7e9a44895662e4993fe2e56e87392db7feba06105561e9693caca615 not found: ID does not exist" containerID="d6019e7b7e9a44895662e4993fe2e56e87392db7feba06105561e9693caca615" Sep 06 06:11:48.918649 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.918597 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6019e7b7e9a44895662e4993fe2e56e87392db7feba06105561e9693caca615"} err="failed to get container status \"d6019e7b7e9a44895662e4993fe2e56e87392db7feba06105561e9693caca615\": rpc error: code = NotFound desc = could not find container \"d6019e7b7e9a44895662e4993fe2e56e87392db7feba06105561e9693caca615\": container with ID starting with d6019e7b7e9a44895662e4993fe2e56e87392db7feba06105561e9693caca615 not found: ID does not exist" Sep 06 06:11:48.919887 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.919870 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh"] Sep 06 06:11:48.923280 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:48.923255 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5da01692-openshift-default-7fcbf75b9b-5jzsh"] Sep 06 06:11:49.151671 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:49.151585 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6256f6d-461c-403f-85f3-686fa2c0e599" path="/var/lib/kubelet/pods/b6256f6d-461c-403f-85f3-686fa2c0e599/volumes" Sep 06 06:11:50.991273 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:11:50.991233 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:12:01.921067 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:01.921028 2578 patch_prober.go:28] interesting pod/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 06 06:12:01.921518 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:01.921098 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" podUID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 06 06:12:02.979527 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:02.979492 2578 generic.go:356] "Generic (PLEG): container finished" podID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" containerID="b95b09d4ea317b245189fc3cbee53a744027679640fdbba74b9c0e92fca3a4a8" exitCode=0 Sep 06 06:12:02.979950 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:02.979565 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" event={"ID":"faee71d7-c3f8-4d18-88d4-e18b492a81a0","Type":"ContainerDied","Data":"b95b09d4ea317b245189fc3cbee53a744027679640fdbba74b9c0e92fca3a4a8"} Sep 06 06:12:02.980004 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:02.979990 2578 scope.go:117] "RemoveContainer" containerID="b95b09d4ea317b245189fc3cbee53a744027679640fdbba74b9c0e92fca3a4a8" Sep 06 06:12:02.981032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:02.981007 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:12:02.981097 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:02.981072 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:12:02.981134 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:02.981123 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:12:03.986876 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:03.986841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" event={"ID":"faee71d7-c3f8-4d18-88d4-e18b492a81a0","Type":"ContainerStarted","Data":"705bc0ee9b479a366fb6a58b710fabcdb4d366ef96ab81faa25f50da3790a0df"} Sep 06 06:12:04.435431 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:04.435394 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:12:04.435611 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:04.435590 2578 patch_prober.go:28] interesting pod/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 06 06:12:04.435659 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:04.435634 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" podUID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 06 06:12:05.435673 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:05.435638 2578 patch_prober.go:28] interesting pod/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 06 06:12:05.436135 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:05.435703 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" podUID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 06 06:12:06.435359 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:06.435321 2578 patch_prober.go:28] interesting pod/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 06 06:12:06.435532 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:06.435397 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" podUID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 06 06:12:06.484138 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:06.484104 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:12:07.435999 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:07.435961 2578 patch_prober.go:28] interesting pod/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 06 06:12:07.436267 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:07.436016 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" podUID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 06 06:12:08.435165 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:08.435124 2578 patch_prober.go:28] interesting pod/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 06 06:12:08.435557 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:08.435196 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" podUID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 06 06:12:09.439416 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:09.439380 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:12:09.439886 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:09.439769 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:12:09.440530 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:09.440510 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:12:11.242229 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:11.242197 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2"] Sep 06 06:12:11.242613 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:11.242590 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerName="istio-proxy" Sep 06 06:12:11.242613 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:11.242601 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerName="istio-proxy" Sep 06 06:12:11.242691 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:11.242684 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerName="istio-proxy" Sep 06 06:12:11.248973 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:11.248951 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2" Sep 06 06:12:11.251155 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:11.251132 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:12:11.251656 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:11.251440 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:12:11.253266 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:11.253247 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2"] Sep 06 06:12:11.388962 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:11.388895 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9452bad-85f2-4f33-b688-bd664e7e7761-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2\" (UID: \"e9452bad-85f2-4f33-b688-bd664e7e7761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2" Sep 06 06:12:11.489821 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:11.489785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9452bad-85f2-4f33-b688-bd664e7e7761-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2\" (UID: \"e9452bad-85f2-4f33-b688-bd664e7e7761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2" Sep 06 06:12:11.490419 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:11.490397 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9452bad-85f2-4f33-b688-bd664e7e7761-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2\" (UID: \"e9452bad-85f2-4f33-b688-bd664e7e7761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2" Sep 06 06:12:11.561694 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:11.561665 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2" Sep 06 06:12:11.830877 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:11.830849 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2"] Sep 06 06:12:11.832592 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:12:11.832565 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9452bad_85f2_4f33_b688_bd664e7e7761.slice/crio-ce3d2f0fe919486555f59600c8fcd4d0936a18c752c525172bd410b687f8a0e9 WatchSource:0}: Error finding container ce3d2f0fe919486555f59600c8fcd4d0936a18c752c525172bd410b687f8a0e9: Status 404 returned error can't find the container with id ce3d2f0fe919486555f59600c8fcd4d0936a18c752c525172bd410b687f8a0e9 Sep 06 06:12:12.035735 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:12.035701 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2" event={"ID":"e9452bad-85f2-4f33-b688-bd664e7e7761","Type":"ContainerStarted","Data":"ce3d2f0fe919486555f59600c8fcd4d0936a18c752c525172bd410b687f8a0e9"} Sep 06 06:12:13.042437 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:13.042405 2578 generic.go:356] "Generic (PLEG): container finished" podID="e9452bad-85f2-4f33-b688-bd664e7e7761" containerID="35adf8f19fdb7358ea3c278a0704502af3fcda35e4c456f8a4190c83c756bf9d" exitCode=0 Sep 06 06:12:13.042841 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:13.042493 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2" event={"ID":"e9452bad-85f2-4f33-b688-bd664e7e7761","Type":"ContainerDied","Data":"35adf8f19fdb7358ea3c278a0704502af3fcda35e4c456f8a4190c83c756bf9d"} Sep 06 06:12:14.191294 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:14.191272 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2" Sep 06 06:12:14.316668 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:14.316589 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9452bad-85f2-4f33-b688-bd664e7e7761-maas-api-service-ca\") pod \"e9452bad-85f2-4f33-b688-bd664e7e7761\" (UID: \"e9452bad-85f2-4f33-b688-bd664e7e7761\") " Sep 06 06:12:14.316947 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:14.316927 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9452bad-85f2-4f33-b688-bd664e7e7761-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9452bad-85f2-4f33-b688-bd664e7e7761" (UID: "e9452bad-85f2-4f33-b688-bd664e7e7761"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:12:14.418257 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:14.418225 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9452bad-85f2-4f33-b688-bd664e7e7761-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:15.054654 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:15.054624 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2" Sep 06 06:12:15.054842 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:15.054667 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2" event={"ID":"e9452bad-85f2-4f33-b688-bd664e7e7761","Type":"ContainerDied","Data":"ce3d2f0fe919486555f59600c8fcd4d0936a18c752c525172bd410b687f8a0e9"} Sep 06 06:12:15.054842 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:15.054709 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce3d2f0fe919486555f59600c8fcd4d0936a18c752c525172bd410b687f8a0e9" Sep 06 06:12:18.134737 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:18.134704 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2"] Sep 06 06:12:18.139036 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:18.139013 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06e64f94-vd9j2"] Sep 06 06:12:18.266973 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:18.266939 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg"] Sep 06 06:12:18.267245 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:18.267195 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" podUID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" containerName="istio-proxy" containerID="cri-o://705bc0ee9b479a366fb6a58b710fabcdb4d366ef96ab81faa25f50da3790a0df" gracePeriod=30 Sep 06 06:12:18.982061 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:18.982029 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5"] Sep 06 06:12:18.982460 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:18.982446 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerName="istio-proxy" Sep 06 06:12:18.982511 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:18.982461 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerName="istio-proxy" Sep 06 06:12:18.982511 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:18.982501 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9452bad-85f2-4f33-b688-bd664e7e7761" containerName="revoke-keys" Sep 06 06:12:18.982511 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:18.982506 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9452bad-85f2-4f33-b688-bd664e7e7761" containerName="revoke-keys" Sep 06 06:12:18.982621 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:18.982567 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6256f6d-461c-403f-85f3-686fa2c0e599" containerName="istio-proxy" Sep 06 06:12:18.982621 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:18.982577 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9452bad-85f2-4f33-b688-bd664e7e7761" containerName="revoke-keys" Sep 06 06:12:18.985851 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:18.985835 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:18.988137 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:18.988113 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-efdea9ac-openshift-default-dockercfg-tsxb5\"" Sep 06 06:12:18.996875 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:18.996854 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5"] Sep 06 06:12:19.063057 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.063025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-workload-socket\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.063216 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.063079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvfmj\" (UniqueName: \"kubernetes.io/projected/29ab0249-1375-416c-906b-8ee4a6cd2dd9-kube-api-access-nvfmj\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.063216 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.063150 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-credential-socket\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.063216 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.063192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istiod-ca-cert\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.063216 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.063213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-podinfo\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.063438 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.063234 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-workload-certs\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.063438 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.063327 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-token\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.063438 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.063365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-data\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.063560 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.063441 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-envoy\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.161222 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.161179 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9452bad-85f2-4f33-b688-bd664e7e7761" path="/var/lib/kubelet/pods/e9452bad-85f2-4f33-b688-bd664e7e7761/volumes" Sep 06 06:12:19.164735 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.164709 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-envoy\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.164864 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.164760 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-workload-socket\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.164864 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.164796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvfmj\" (UniqueName: \"kubernetes.io/projected/29ab0249-1375-416c-906b-8ee4a6cd2dd9-kube-api-access-nvfmj\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.164864 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.164817 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-credential-socket\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.164864 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.164839 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istiod-ca-cert\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.164864 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.164856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-podinfo\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.165056 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.164872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-workload-certs\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.165056 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.164918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-token\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.165056 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.164937 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-data\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.165233 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.165218 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-data\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.167586 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.167556 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-envoy\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.167821 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.167797 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-workload-socket\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.168360 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.168340 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-credential-socket\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.169173 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.169151 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istiod-ca-cert\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.169786 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.169762 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-workload-certs\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.173577 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.173535 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-podinfo\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.176778 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.176719 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvfmj\" (UniqueName: \"kubernetes.io/projected/29ab0249-1375-416c-906b-8ee4a6cd2dd9-kube-api-access-nvfmj\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.177846 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.177825 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-token\") pod \"e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.297322 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.297284 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:19.580785 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.580758 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5"] Sep 06 06:12:19.582643 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:12:19.582611 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ab0249_1375_416c_906b_8ee4a6cd2dd9.slice/crio-36e9aca97f2512dfee7f614394c0f33d8ea26dbdffab77478a3ecfcdc6e32013 WatchSource:0}: Error finding container 36e9aca97f2512dfee7f614394c0f33d8ea26dbdffab77478a3ecfcdc6e32013: Status 404 returned error can't find the container with id 36e9aca97f2512dfee7f614394c0f33d8ea26dbdffab77478a3ecfcdc6e32013 Sep 06 06:12:19.584963 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.584932 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:12:19.585038 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.584999 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:12:19.585038 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:19.585027 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:12:20.087879 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:20.087836 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" event={"ID":"29ab0249-1375-416c-906b-8ee4a6cd2dd9","Type":"ContainerStarted","Data":"62c1d0ed45374e978469ed5e2195d64e7654bd5917cbff7542fe1f17cb50c86f"} Sep 06 06:12:20.088112 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:20.087893 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" event={"ID":"29ab0249-1375-416c-906b-8ee4a6cd2dd9","Type":"ContainerStarted","Data":"36e9aca97f2512dfee7f614394c0f33d8ea26dbdffab77478a3ecfcdc6e32013"} Sep 06 06:12:20.108722 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:20.108671 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" podStartSLOduration=2.108655391 podStartE2EDuration="2.108655391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:12:20.106687359 +0000 UTC m=+1105.505017844" watchObservedRunningTime="2026-09-06 06:12:20.108655391 +0000 UTC m=+1105.506985918" Sep 06 06:12:20.297489 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:20.297458 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:20.302274 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:20.302250 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:21.094967 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:21.094934 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:21.096026 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:21.096003 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:22.292396 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:22.292361 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:12:23.530766 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.530709 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:12:23.610835 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.609240 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv84g\" (UniqueName: \"kubernetes.io/projected/faee71d7-c3f8-4d18-88d4-e18b492a81a0-kube-api-access-nv84g\") pod \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " Sep 06 06:12:23.610835 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.609343 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-credential-socket\") pod \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " Sep 06 06:12:23.610835 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.609384 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-workload-socket\") pod \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " Sep 06 06:12:23.610835 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.609436 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istiod-ca-cert\") pod \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " Sep 06 06:12:23.610835 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.609490 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-workload-certs\") pod \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " Sep 06 06:12:23.610835 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.609560 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-envoy\") pod \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " Sep 06 06:12:23.610835 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.609609 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-data\") pod \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " Sep 06 06:12:23.610835 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.609660 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-token\") pod \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " Sep 06 06:12:23.610835 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.609691 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-podinfo\") pod \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\" (UID: \"faee71d7-c3f8-4d18-88d4-e18b492a81a0\") " Sep 06 06:12:23.615334 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.615296 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "faee71d7-c3f8-4d18-88d4-e18b492a81a0" (UID: "faee71d7-c3f8-4d18-88d4-e18b492a81a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:12:23.616718 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.616691 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "faee71d7-c3f8-4d18-88d4-e18b492a81a0" (UID: "faee71d7-c3f8-4d18-88d4-e18b492a81a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:12:23.616975 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.616929 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "faee71d7-c3f8-4d18-88d4-e18b492a81a0" (UID: "faee71d7-c3f8-4d18-88d4-e18b492a81a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:12:23.617065 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.616706 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "faee71d7-c3f8-4d18-88d4-e18b492a81a0" (UID: "faee71d7-c3f8-4d18-88d4-e18b492a81a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:12:23.617065 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.617049 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-data" (OuterVolumeSpecName: "istio-data") pod "faee71d7-c3f8-4d18-88d4-e18b492a81a0" (UID: "faee71d7-c3f8-4d18-88d4-e18b492a81a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:12:23.618309 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.618280 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faee71d7-c3f8-4d18-88d4-e18b492a81a0-kube-api-access-nv84g" (OuterVolumeSpecName: "kube-api-access-nv84g") pod "faee71d7-c3f8-4d18-88d4-e18b492a81a0" (UID: "faee71d7-c3f8-4d18-88d4-e18b492a81a0"). InnerVolumeSpecName "kube-api-access-nv84g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:12:23.619051 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.619020 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "faee71d7-c3f8-4d18-88d4-e18b492a81a0" (UID: "faee71d7-c3f8-4d18-88d4-e18b492a81a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:12:23.619168 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.619135 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "faee71d7-c3f8-4d18-88d4-e18b492a81a0" (UID: "faee71d7-c3f8-4d18-88d4-e18b492a81a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:12:23.619350 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.619320 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-token" (OuterVolumeSpecName: "istio-token") pod "faee71d7-c3f8-4d18-88d4-e18b492a81a0" (UID: "faee71d7-c3f8-4d18-88d4-e18b492a81a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:12:23.711238 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.711200 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:23.711238 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.711233 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:23.711238 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.711241 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:23.711465 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.711248 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:23.711465 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.711256 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:23.711465 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.711265 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv84g\" (UniqueName: \"kubernetes.io/projected/faee71d7-c3f8-4d18-88d4-e18b492a81a0-kube-api-access-nv84g\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:23.711465 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.711276 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:23.711465 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.711283 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faee71d7-c3f8-4d18-88d4-e18b492a81a0-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:23.711465 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:23.711292 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faee71d7-c3f8-4d18-88d4-e18b492a81a0-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:24.117275 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:24.117245 2578 generic.go:356] "Generic (PLEG): container finished" podID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" containerID="705bc0ee9b479a366fb6a58b710fabcdb4d366ef96ab81faa25f50da3790a0df" exitCode=0 Sep 06 06:12:24.117497 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:24.117320 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" event={"ID":"faee71d7-c3f8-4d18-88d4-e18b492a81a0","Type":"ContainerDied","Data":"705bc0ee9b479a366fb6a58b710fabcdb4d366ef96ab81faa25f50da3790a0df"} Sep 06 06:12:24.117497 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:24.117332 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" Sep 06 06:12:24.117497 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:24.117425 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg" event={"ID":"faee71d7-c3f8-4d18-88d4-e18b492a81a0","Type":"ContainerDied","Data":"43cdd75c53f56391520ecb79e4555046d1c988abf1c23ec02f7f62884f760fc7"} Sep 06 06:12:24.117497 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:24.117457 2578 scope.go:117] "RemoveContainer" containerID="705bc0ee9b479a366fb6a58b710fabcdb4d366ef96ab81faa25f50da3790a0df" Sep 06 06:12:24.127350 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:24.127336 2578 scope.go:117] "RemoveContainer" containerID="b95b09d4ea317b245189fc3cbee53a744027679640fdbba74b9c0e92fca3a4a8" Sep 06 06:12:24.136031 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:24.136013 2578 scope.go:117] "RemoveContainer" containerID="705bc0ee9b479a366fb6a58b710fabcdb4d366ef96ab81faa25f50da3790a0df" Sep 06 06:12:24.136284 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:12:24.136266 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705bc0ee9b479a366fb6a58b710fabcdb4d366ef96ab81faa25f50da3790a0df\": container with ID starting with 705bc0ee9b479a366fb6a58b710fabcdb4d366ef96ab81faa25f50da3790a0df not found: ID does not exist" containerID="705bc0ee9b479a366fb6a58b710fabcdb4d366ef96ab81faa25f50da3790a0df" Sep 06 06:12:24.136349 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:24.136290 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705bc0ee9b479a366fb6a58b710fabcdb4d366ef96ab81faa25f50da3790a0df"} err="failed to get container status \"705bc0ee9b479a366fb6a58b710fabcdb4d366ef96ab81faa25f50da3790a0df\": rpc error: code = NotFound desc = could not find container \"705bc0ee9b479a366fb6a58b710fabcdb4d366ef96ab81faa25f50da3790a0df\": container with ID starting with 705bc0ee9b479a366fb6a58b710fabcdb4d366ef96ab81faa25f50da3790a0df not found: ID does not exist" Sep 06 06:12:24.136349 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:24.136306 2578 scope.go:117] "RemoveContainer" containerID="b95b09d4ea317b245189fc3cbee53a744027679640fdbba74b9c0e92fca3a4a8" Sep 06 06:12:24.136518 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:12:24.136503 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95b09d4ea317b245189fc3cbee53a744027679640fdbba74b9c0e92fca3a4a8\": container with ID starting with b95b09d4ea317b245189fc3cbee53a744027679640fdbba74b9c0e92fca3a4a8 not found: ID does not exist" containerID="b95b09d4ea317b245189fc3cbee53a744027679640fdbba74b9c0e92fca3a4a8" Sep 06 06:12:24.136557 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:24.136522 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95b09d4ea317b245189fc3cbee53a744027679640fdbba74b9c0e92fca3a4a8"} err="failed to get container status \"b95b09d4ea317b245189fc3cbee53a744027679640fdbba74b9c0e92fca3a4a8\": rpc error: code = NotFound desc = could not find container \"b95b09d4ea317b245189fc3cbee53a744027679640fdbba74b9c0e92fca3a4a8\": container with ID starting with b95b09d4ea317b245189fc3cbee53a744027679640fdbba74b9c0e92fca3a4a8 not found: ID does not exist" Sep 06 06:12:24.140749 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:24.140729 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg"] Sep 06 06:12:24.143669 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:24.143650 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-06e64f94-gw-openshift-default-5fbc5c8b8d-r2zkg"] Sep 06 06:12:25.144973 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:25.144936 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" path="/var/lib/kubelet/pods/faee71d7-c3f8-4d18-88d4-e18b492a81a0/volumes" Sep 06 06:12:37.593101 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:37.593065 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:12:42.229319 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:42.229281 2578 generic.go:356] "Generic (PLEG): container finished" podID="29ab0249-1375-416c-906b-8ee4a6cd2dd9" containerID="62c1d0ed45374e978469ed5e2195d64e7654bd5917cbff7542fe1f17cb50c86f" exitCode=0 Sep 06 06:12:42.229735 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:42.229348 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" event={"ID":"29ab0249-1375-416c-906b-8ee4a6cd2dd9","Type":"ContainerDied","Data":"62c1d0ed45374e978469ed5e2195d64e7654bd5917cbff7542fe1f17cb50c86f"} Sep 06 06:12:42.229780 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:42.229745 2578 scope.go:117] "RemoveContainer" containerID="62c1d0ed45374e978469ed5e2195d64e7654bd5917cbff7542fe1f17cb50c86f" Sep 06 06:12:42.230895 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:42.230861 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:12:42.230975 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:42.230961 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:12:42.231012 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:42.230995 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:12:43.237103 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:43.237066 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" event={"ID":"29ab0249-1375-416c-906b-8ee4a6cd2dd9","Type":"ContainerStarted","Data":"20c502c34e10c82f00d67befe166f2256717d5b604c8cae8c3dcb320c432103e"} Sep 06 06:12:43.298140 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:43.298100 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:43.298322 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:43.298299 2578 patch_prober.go:28] interesting pod/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 06 06:12:43.298368 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:43.298348 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" podUID="29ab0249-1375-416c-906b-8ee4a6cd2dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 06 06:12:44.298272 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:44.298226 2578 patch_prober.go:28] interesting pod/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 06 06:12:44.298661 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:44.298304 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" podUID="29ab0249-1375-416c-906b-8ee4a6cd2dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 06 06:12:45.297860 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:45.297815 2578 patch_prober.go:28] interesting pod/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 06 06:12:45.298102 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:45.297877 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" podUID="29ab0249-1375-416c-906b-8ee4a6cd2dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 06 06:12:46.297767 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:46.297733 2578 patch_prober.go:28] interesting pod/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 06 06:12:46.298182 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:46.297788 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" podUID="29ab0249-1375-416c-906b-8ee4a6cd2dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 06 06:12:47.106475 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.106428 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56"] Sep 06 06:12:47.106945 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.106930 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" containerName="istio-proxy" Sep 06 06:12:47.107010 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.106947 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" containerName="istio-proxy" Sep 06 06:12:47.107010 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.106981 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" containerName="istio-proxy" Sep 06 06:12:47.107010 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.106987 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" containerName="istio-proxy" Sep 06 06:12:47.107104 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.107062 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" containerName="istio-proxy" Sep 06 06:12:47.107104 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.107070 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="faee71d7-c3f8-4d18-88d4-e18b492a81a0" containerName="istio-proxy" Sep 06 06:12:47.110237 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.110220 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56" Sep 06 06:12:47.112394 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.112372 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:12:47.112394 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.112386 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:12:47.117088 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.117059 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56"] Sep 06 06:12:47.270476 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.270430 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78797db-9b9e-4864-b383-fc47d7838fda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56\" (UID: \"b78797db-9b9e-4864-b383-fc47d7838fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56" Sep 06 06:12:47.298247 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.298208 2578 patch_prober.go:28] interesting pod/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 06 06:12:47.298741 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.298286 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" podUID="29ab0249-1375-416c-906b-8ee4a6cd2dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 06 06:12:47.371402 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.371304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78797db-9b9e-4864-b383-fc47d7838fda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56\" (UID: \"b78797db-9b9e-4864-b383-fc47d7838fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56" Sep 06 06:12:47.372256 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.372226 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78797db-9b9e-4864-b383-fc47d7838fda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56\" (UID: \"b78797db-9b9e-4864-b383-fc47d7838fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56" Sep 06 06:12:47.424410 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.424363 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56" Sep 06 06:12:47.737420 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:47.737390 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56"] Sep 06 06:12:47.739445 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:12:47.739409 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78797db_9b9e_4864_b383_fc47d7838fda.slice/crio-f0b880b296925841491b0ea9ef80c4965651c040b0d998feff756a91aba695fc WatchSource:0}: Error finding container f0b880b296925841491b0ea9ef80c4965651c040b0d998feff756a91aba695fc: Status 404 returned error can't find the container with id f0b880b296925841491b0ea9ef80c4965651c040b0d998feff756a91aba695fc Sep 06 06:12:48.276161 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:48.275297 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56" event={"ID":"b78797db-9b9e-4864-b383-fc47d7838fda","Type":"ContainerStarted","Data":"19b02b83f874dc890b07d2319d430db4f52b579a0b4b1014ba037f46b1e257dd"} Sep 06 06:12:48.276161 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:48.275363 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56" event={"ID":"b78797db-9b9e-4864-b383-fc47d7838fda","Type":"ContainerStarted","Data":"f0b880b296925841491b0ea9ef80c4965651c040b0d998feff756a91aba695fc"} Sep 06 06:12:48.290246 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:48.290195 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56" podStartSLOduration=1.290178176 podStartE2EDuration="1.290178176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:12:48.288965639 +0000 UTC m=+1133.687296121" watchObservedRunningTime="2026-09-06 06:12:48.290178176 +0000 UTC m=+1133.688508643" Sep 06 06:12:48.419872 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:48.419773 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:48.420557 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:48.420508 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:48.421777 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:48.421753 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:49.282351 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:49.282313 2578 generic.go:356] "Generic (PLEG): container finished" podID="b78797db-9b9e-4864-b383-fc47d7838fda" containerID="19b02b83f874dc890b07d2319d430db4f52b579a0b4b1014ba037f46b1e257dd" exitCode=0 Sep 06 06:12:49.282508 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:49.282402 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56" event={"ID":"b78797db-9b9e-4864-b383-fc47d7838fda","Type":"ContainerDied","Data":"19b02b83f874dc890b07d2319d430db4f52b579a0b4b1014ba037f46b1e257dd"} Sep 06 06:12:50.431186 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:50.431160 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56" Sep 06 06:12:50.614399 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:50.614310 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78797db-9b9e-4864-b383-fc47d7838fda-maas-api-service-ca\") pod \"b78797db-9b9e-4864-b383-fc47d7838fda\" (UID: \"b78797db-9b9e-4864-b383-fc47d7838fda\") " Sep 06 06:12:50.614697 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:50.614671 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78797db-9b9e-4864-b383-fc47d7838fda-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b78797db-9b9e-4864-b383-fc47d7838fda" (UID: "b78797db-9b9e-4864-b383-fc47d7838fda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:12:50.716011 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:50.715967 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78797db-9b9e-4864-b383-fc47d7838fda-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:51.294500 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:51.294469 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56" Sep 06 06:12:51.294667 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:51.294497 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56" event={"ID":"b78797db-9b9e-4864-b383-fc47d7838fda","Type":"ContainerDied","Data":"f0b880b296925841491b0ea9ef80c4965651c040b0d998feff756a91aba695fc"} Sep 06 06:12:51.294667 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:51.294547 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0b880b296925841491b0ea9ef80c4965651c040b0d998feff756a91aba695fc" Sep 06 06:12:52.985618 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:52.985579 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:12:53.447079 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:53.447045 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56"] Sep 06 06:12:53.449007 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:53.448987 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-efdea9ac-qwd56"] Sep 06 06:12:54.541267 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:54.541224 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5"] Sep 06 06:12:54.541758 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:54.541432 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" podUID="29ab0249-1375-416c-906b-8ee4a6cd2dd9" containerName="istio-proxy" containerID="cri-o://20c502c34e10c82f00d67befe166f2256717d5b604c8cae8c3dcb320c432103e" gracePeriod=30 Sep 06 06:12:55.145596 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.145548 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b78797db-9b9e-4864-b383-fc47d7838fda" path="/var/lib/kubelet/pods/b78797db-9b9e-4864-b383-fc47d7838fda/volumes" Sep 06 06:12:55.223458 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.223422 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq"] Sep 06 06:12:55.223818 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.223805 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b78797db-9b9e-4864-b383-fc47d7838fda" containerName="revoke-keys" Sep 06 06:12:55.223858 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.223820 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b78797db-9b9e-4864-b383-fc47d7838fda" containerName="revoke-keys" Sep 06 06:12:55.223929 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.223896 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b78797db-9b9e-4864-b383-fc47d7838fda" containerName="revoke-keys" Sep 06 06:12:55.228212 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.228196 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.230196 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.230170 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f260fc83-openshift-default-dockercfg-5qlw7\"" Sep 06 06:12:55.239188 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.239159 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq"] Sep 06 06:12:55.260876 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.260841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f02530e-a95c-413a-a77a-11760ac19203-istiod-ca-cert\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.261089 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.260889 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-istio-data\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.261089 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.260927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f02530e-a95c-413a-a77a-11760ac19203-istio-token\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.261089 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.260951 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-istio-envoy\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.261089 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.261019 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npjw6\" (UniqueName: \"kubernetes.io/projected/4f02530e-a95c-413a-a77a-11760ac19203-kube-api-access-npjw6\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.261343 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.261097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f02530e-a95c-413a-a77a-11760ac19203-istio-podinfo\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.261343 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.261144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-workload-socket\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.261343 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.261220 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-credential-socket\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.261343 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.261251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-workload-certs\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.362267 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.362230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-workload-socket\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.362462 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.362283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-credential-socket\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.362462 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.362404 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-workload-certs\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.362589 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.362487 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f02530e-a95c-413a-a77a-11760ac19203-istiod-ca-cert\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.362589 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.362526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-istio-data\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.362589 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.362564 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f02530e-a95c-413a-a77a-11760ac19203-istio-token\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.362589 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.362587 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-credential-socket\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.362802 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.362603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-istio-envoy\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.362802 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.362643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npjw6\" (UniqueName: \"kubernetes.io/projected/4f02530e-a95c-413a-a77a-11760ac19203-kube-api-access-npjw6\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.362802 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.362679 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-workload-socket\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.362802 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.362702 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f02530e-a95c-413a-a77a-11760ac19203-istio-podinfo\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.362802 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.362730 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-workload-certs\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.363047 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.362805 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-istio-data\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.363222 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.363203 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f02530e-a95c-413a-a77a-11760ac19203-istiod-ca-cert\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.365001 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.364976 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-istio-envoy\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.365222 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.365206 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f02530e-a95c-413a-a77a-11760ac19203-istio-podinfo\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.369934 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.369891 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f02530e-a95c-413a-a77a-11760ac19203-istio-token\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.370206 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.370187 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npjw6\" (UniqueName: \"kubernetes.io/projected/4f02530e-a95c-413a-a77a-11760ac19203-kube-api-access-npjw6\") pod \"e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.540173 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.540139 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:55.831587 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.831559 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq"] Sep 06 06:12:55.833089 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:12:55.833059 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f02530e_a95c_413a_a77a_11760ac19203.slice/crio-edebec35c57407efb9553e0833971821044db4f144781d074a0a953059df91c3 WatchSource:0}: Error finding container edebec35c57407efb9553e0833971821044db4f144781d074a0a953059df91c3: Status 404 returned error can't find the container with id edebec35c57407efb9553e0833971821044db4f144781d074a0a953059df91c3 Sep 06 06:12:55.835732 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.835688 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:12:55.835808 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.835786 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:12:55.835847 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:55.835829 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:12:56.325127 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:56.325085 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" event={"ID":"4f02530e-a95c-413a-a77a-11760ac19203","Type":"ContainerStarted","Data":"2ef333a3f526265f2c863a8aecb7721c2e4d6ec1f7f676518334def20a0d79c2"} Sep 06 06:12:56.325291 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:56.325150 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" event={"ID":"4f02530e-a95c-413a-a77a-11760ac19203","Type":"ContainerStarted","Data":"edebec35c57407efb9553e0833971821044db4f144781d074a0a953059df91c3"} Sep 06 06:12:56.345720 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:56.345670 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" podStartSLOduration=1.345652371 podStartE2EDuration="1.345652371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:12:56.342731274 +0000 UTC m=+1141.741061742" watchObservedRunningTime="2026-09-06 06:12:56.345652371 +0000 UTC m=+1141.743982839" Sep 06 06:12:56.540559 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:56.540523 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:56.545757 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:56.545731 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:57.330255 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:57.330217 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:57.331320 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:57.331298 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:12:59.803816 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.803759 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:12:59.918002 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.917963 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-workload-socket\") pod \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " Sep 06 06:12:59.918002 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918002 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-workload-certs\") pod \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " Sep 06 06:12:59.918259 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918047 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-data\") pod \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " Sep 06 06:12:59.918259 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918071 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-token\") pod \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " Sep 06 06:12:59.918259 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918121 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvfmj\" (UniqueName: \"kubernetes.io/projected/29ab0249-1375-416c-906b-8ee4a6cd2dd9-kube-api-access-nvfmj\") pod \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " Sep 06 06:12:59.918259 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918170 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-envoy\") pod \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " Sep 06 06:12:59.918259 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918211 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istiod-ca-cert\") pod \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " Sep 06 06:12:59.918524 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918253 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "29ab0249-1375-416c-906b-8ee4a6cd2dd9" (UID: "29ab0249-1375-416c-906b-8ee4a6cd2dd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:12:59.918524 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918264 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-podinfo\") pod \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " Sep 06 06:12:59.918524 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918356 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-data" (OuterVolumeSpecName: "istio-data") pod "29ab0249-1375-416c-906b-8ee4a6cd2dd9" (UID: "29ab0249-1375-416c-906b-8ee4a6cd2dd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:12:59.918524 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918374 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "29ab0249-1375-416c-906b-8ee4a6cd2dd9" (UID: "29ab0249-1375-416c-906b-8ee4a6cd2dd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:12:59.918524 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918388 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-credential-socket\") pod \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\" (UID: \"29ab0249-1375-416c-906b-8ee4a6cd2dd9\") " Sep 06 06:12:59.918776 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918687 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "29ab0249-1375-416c-906b-8ee4a6cd2dd9" (UID: "29ab0249-1375-416c-906b-8ee4a6cd2dd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:12:59.918776 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918727 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "29ab0249-1375-416c-906b-8ee4a6cd2dd9" (UID: "29ab0249-1375-416c-906b-8ee4a6cd2dd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:12:59.918874 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918832 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:59.918874 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918851 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:59.918874 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918866 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:59.919028 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918879 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:59.919028 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.918893 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:12:59.920493 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.920469 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ab0249-1375-416c-906b-8ee4a6cd2dd9-kube-api-access-nvfmj" (OuterVolumeSpecName: "kube-api-access-nvfmj") pod "29ab0249-1375-416c-906b-8ee4a6cd2dd9" (UID: "29ab0249-1375-416c-906b-8ee4a6cd2dd9"). InnerVolumeSpecName "kube-api-access-nvfmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:12:59.920598 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.920558 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "29ab0249-1375-416c-906b-8ee4a6cd2dd9" (UID: "29ab0249-1375-416c-906b-8ee4a6cd2dd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:12:59.921010 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.920984 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-token" (OuterVolumeSpecName: "istio-token") pod "29ab0249-1375-416c-906b-8ee4a6cd2dd9" (UID: "29ab0249-1375-416c-906b-8ee4a6cd2dd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:12:59.921118 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:12:59.921043 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "29ab0249-1375-416c-906b-8ee4a6cd2dd9" (UID: "29ab0249-1375-416c-906b-8ee4a6cd2dd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:13:00.020249 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.020212 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:00.020249 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.020242 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:00.020249 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.020251 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ab0249-1375-416c-906b-8ee4a6cd2dd9-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:00.020249 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.020261 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvfmj\" (UniqueName: \"kubernetes.io/projected/29ab0249-1375-416c-906b-8ee4a6cd2dd9-kube-api-access-nvfmj\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:00.348332 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.348295 2578 generic.go:356] "Generic (PLEG): container finished" podID="29ab0249-1375-416c-906b-8ee4a6cd2dd9" containerID="20c502c34e10c82f00d67befe166f2256717d5b604c8cae8c3dcb320c432103e" exitCode=0 Sep 06 06:13:00.348501 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.348345 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" Sep 06 06:13:00.348501 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.348376 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" event={"ID":"29ab0249-1375-416c-906b-8ee4a6cd2dd9","Type":"ContainerDied","Data":"20c502c34e10c82f00d67befe166f2256717d5b604c8cae8c3dcb320c432103e"} Sep 06 06:13:00.348501 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.348433 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5" event={"ID":"29ab0249-1375-416c-906b-8ee4a6cd2dd9","Type":"ContainerDied","Data":"36e9aca97f2512dfee7f614394c0f33d8ea26dbdffab77478a3ecfcdc6e32013"} Sep 06 06:13:00.348501 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.348457 2578 scope.go:117] "RemoveContainer" containerID="20c502c34e10c82f00d67befe166f2256717d5b604c8cae8c3dcb320c432103e" Sep 06 06:13:00.358932 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.358890 2578 scope.go:117] "RemoveContainer" containerID="62c1d0ed45374e978469ed5e2195d64e7654bd5917cbff7542fe1f17cb50c86f" Sep 06 06:13:00.367861 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.367841 2578 scope.go:117] "RemoveContainer" containerID="20c502c34e10c82f00d67befe166f2256717d5b604c8cae8c3dcb320c432103e" Sep 06 06:13:00.368218 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:13:00.368195 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c502c34e10c82f00d67befe166f2256717d5b604c8cae8c3dcb320c432103e\": container with ID starting with 20c502c34e10c82f00d67befe166f2256717d5b604c8cae8c3dcb320c432103e not found: ID does not exist" containerID="20c502c34e10c82f00d67befe166f2256717d5b604c8cae8c3dcb320c432103e" Sep 06 06:13:00.368292 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.368227 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c502c34e10c82f00d67befe166f2256717d5b604c8cae8c3dcb320c432103e"} err="failed to get container status \"20c502c34e10c82f00d67befe166f2256717d5b604c8cae8c3dcb320c432103e\": rpc error: code = NotFound desc = could not find container \"20c502c34e10c82f00d67befe166f2256717d5b604c8cae8c3dcb320c432103e\": container with ID starting with 20c502c34e10c82f00d67befe166f2256717d5b604c8cae8c3dcb320c432103e not found: ID does not exist" Sep 06 06:13:00.368292 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.368247 2578 scope.go:117] "RemoveContainer" containerID="62c1d0ed45374e978469ed5e2195d64e7654bd5917cbff7542fe1f17cb50c86f" Sep 06 06:13:00.368529 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:13:00.368507 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c1d0ed45374e978469ed5e2195d64e7654bd5917cbff7542fe1f17cb50c86f\": container with ID starting with 62c1d0ed45374e978469ed5e2195d64e7654bd5917cbff7542fe1f17cb50c86f not found: ID does not exist" containerID="62c1d0ed45374e978469ed5e2195d64e7654bd5917cbff7542fe1f17cb50c86f" Sep 06 06:13:00.368585 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.368536 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c1d0ed45374e978469ed5e2195d64e7654bd5917cbff7542fe1f17cb50c86f"} err="failed to get container status \"62c1d0ed45374e978469ed5e2195d64e7654bd5917cbff7542fe1f17cb50c86f\": rpc error: code = NotFound desc = could not find container \"62c1d0ed45374e978469ed5e2195d64e7654bd5917cbff7542fe1f17cb50c86f\": container with ID starting with 62c1d0ed45374e978469ed5e2195d64e7654bd5917cbff7542fe1f17cb50c86f not found: ID does not exist" Sep 06 06:13:00.370018 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.369981 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5"] Sep 06 06:13:00.372936 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:00.372888 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-efdea9ac-openshift-default-5d685dcbbd-qf9g5"] Sep 06 06:13:01.145823 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:01.145783 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29ab0249-1375-416c-906b-8ee4a6cd2dd9" path="/var/lib/kubelet/pods/29ab0249-1375-416c-906b-8ee4a6cd2dd9/volumes" Sep 06 06:13:13.371107 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:13.371054 2578 patch_prober.go:28] interesting pod/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 06 06:13:13.371667 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:13.371144 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" podUID="4f02530e-a95c-413a-a77a-11760ac19203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 06 06:13:15.292574 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:15.292482 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn"] Sep 06 06:13:15.456497 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:15.456461 2578 generic.go:356] "Generic (PLEG): container finished" podID="4f02530e-a95c-413a-a77a-11760ac19203" containerID="2ef333a3f526265f2c863a8aecb7721c2e4d6ec1f7f676518334def20a0d79c2" exitCode=0 Sep 06 06:13:15.456669 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:15.456533 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" event={"ID":"4f02530e-a95c-413a-a77a-11760ac19203","Type":"ContainerDied","Data":"2ef333a3f526265f2c863a8aecb7721c2e4d6ec1f7f676518334def20a0d79c2"} Sep 06 06:13:15.457012 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:15.456994 2578 scope.go:117] "RemoveContainer" containerID="2ef333a3f526265f2c863a8aecb7721c2e4d6ec1f7f676518334def20a0d79c2" Sep 06 06:13:15.458174 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:15.458144 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:13:15.458275 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:15.458239 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:13:15.458341 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:15.458292 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:13:16.400561 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:16.400468 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:13:16.469482 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:16.469432 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" event={"ID":"4f02530e-a95c-413a-a77a-11760ac19203","Type":"ContainerStarted","Data":"c7d9c6e58c1c50ab08835268cbe1616d6bec99458462952676e662e6dcfd8a98"} Sep 06 06:13:16.540853 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:16.540818 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:13:16.541064 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:16.541042 2578 patch_prober.go:28] interesting pod/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 06 06:13:16.541116 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:16.541096 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" podUID="4f02530e-a95c-413a-a77a-11760ac19203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 06 06:13:17.438985 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.438943 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54"] Sep 06 06:13:17.439495 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.439481 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29ab0249-1375-416c-906b-8ee4a6cd2dd9" containerName="istio-proxy" Sep 06 06:13:17.439542 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.439506 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="29ab0249-1375-416c-906b-8ee4a6cd2dd9" containerName="istio-proxy" Sep 06 06:13:17.439542 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.439515 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29ab0249-1375-416c-906b-8ee4a6cd2dd9" containerName="istio-proxy" Sep 06 06:13:17.439542 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.439520 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="29ab0249-1375-416c-906b-8ee4a6cd2dd9" containerName="istio-proxy" Sep 06 06:13:17.439639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.439594 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="29ab0249-1375-416c-906b-8ee4a6cd2dd9" containerName="istio-proxy" Sep 06 06:13:17.442867 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.442849 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54" Sep 06 06:13:17.445997 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.445977 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:13:17.446177 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.446153 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:13:17.448792 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.448764 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54"] Sep 06 06:13:17.541223 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.541190 2578 patch_prober.go:28] interesting pod/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 06 06:13:17.541401 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.541245 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" podUID="4f02530e-a95c-413a-a77a-11760ac19203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 06 06:13:17.605071 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.605030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf0bcda8-fb40-4404-abe6-434efa57889c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-e75768-zsf54\" (UID: \"bf0bcda8-fb40-4404-abe6-434efa57889c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54" Sep 06 06:13:17.706355 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.706265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf0bcda8-fb40-4404-abe6-434efa57889c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-e75768-zsf54\" (UID: \"bf0bcda8-fb40-4404-abe6-434efa57889c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54" Sep 06 06:13:17.706896 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.706874 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf0bcda8-fb40-4404-abe6-434efa57889c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-e75768-zsf54\" (UID: \"bf0bcda8-fb40-4404-abe6-434efa57889c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54" Sep 06 06:13:17.760420 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:17.760380 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54" Sep 06 06:13:18.041701 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:18.041670 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54"] Sep 06 06:13:18.046683 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:13:18.046635 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0bcda8_fb40_4404_abe6_434efa57889c.slice/crio-de649cd80b4b2efe7acbb1c8c94d05768e05ebd6a8324f2e29a7cfc437aef94d WatchSource:0}: Error finding container de649cd80b4b2efe7acbb1c8c94d05768e05ebd6a8324f2e29a7cfc437aef94d: Status 404 returned error can't find the container with id de649cd80b4b2efe7acbb1c8c94d05768e05ebd6a8324f2e29a7cfc437aef94d Sep 06 06:13:18.490852 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:18.490821 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54" event={"ID":"bf0bcda8-fb40-4404-abe6-434efa57889c","Type":"ContainerStarted","Data":"96b44c9b8c626f2de63556aa734ef51b304de546948282702d63f25cad39eef9"} Sep 06 06:13:18.491352 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:18.490879 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54" event={"ID":"bf0bcda8-fb40-4404-abe6-434efa57889c","Type":"ContainerStarted","Data":"de649cd80b4b2efe7acbb1c8c94d05768e05ebd6a8324f2e29a7cfc437aef94d"} Sep 06 06:13:18.505503 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:18.505443 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54" podStartSLOduration=1.505426822 podStartE2EDuration="1.505426822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:13:18.504018323 +0000 UTC m=+1163.902348814" watchObservedRunningTime="2026-09-06 06:13:18.505426822 +0000 UTC m=+1163.903757290" Sep 06 06:13:18.541417 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:18.541373 2578 patch_prober.go:28] interesting pod/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 06 06:13:18.541600 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:18.541468 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" podUID="4f02530e-a95c-413a-a77a-11760ac19203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 06 06:13:19.498185 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:19.498031 2578 generic.go:356] "Generic (PLEG): container finished" podID="bf0bcda8-fb40-4404-abe6-434efa57889c" containerID="96b44c9b8c626f2de63556aa734ef51b304de546948282702d63f25cad39eef9" exitCode=0 Sep 06 06:13:19.498185 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:19.498105 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54" event={"ID":"bf0bcda8-fb40-4404-abe6-434efa57889c","Type":"ContainerDied","Data":"96b44c9b8c626f2de63556aa734ef51b304de546948282702d63f25cad39eef9"} Sep 06 06:13:19.541394 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:19.541343 2578 patch_prober.go:28] interesting pod/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 06 06:13:19.541585 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:19.541413 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" podUID="4f02530e-a95c-413a-a77a-11760ac19203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 06 06:13:20.540662 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:20.540622 2578 patch_prober.go:28] interesting pod/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 06 06:13:20.541047 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:20.540684 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" podUID="4f02530e-a95c-413a-a77a-11760ac19203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 06 06:13:20.686690 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:20.686663 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54" Sep 06 06:13:20.839253 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:20.839217 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf0bcda8-fb40-4404-abe6-434efa57889c-maas-api-service-ca\") pod \"bf0bcda8-fb40-4404-abe6-434efa57889c\" (UID: \"bf0bcda8-fb40-4404-abe6-434efa57889c\") " Sep 06 06:13:20.839566 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:20.839540 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf0bcda8-fb40-4404-abe6-434efa57889c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bf0bcda8-fb40-4404-abe6-434efa57889c" (UID: "bf0bcda8-fb40-4404-abe6-434efa57889c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:13:20.940823 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:20.940785 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf0bcda8-fb40-4404-abe6-434efa57889c-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:21.511739 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:21.511653 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54" Sep 06 06:13:21.511889 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:21.511650 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54" event={"ID":"bf0bcda8-fb40-4404-abe6-434efa57889c","Type":"ContainerDied","Data":"de649cd80b4b2efe7acbb1c8c94d05768e05ebd6a8324f2e29a7cfc437aef94d"} Sep 06 06:13:21.511889 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:21.511785 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de649cd80b4b2efe7acbb1c8c94d05768e05ebd6a8324f2e29a7cfc437aef94d" Sep 06 06:13:21.540854 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:21.540822 2578 patch_prober.go:28] interesting pod/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 06 06:13:21.541253 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:21.540875 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" podUID="4f02530e-a95c-413a-a77a-11760ac19203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 06 06:13:22.485687 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.485652 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj"] Sep 06 06:13:22.486158 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.486142 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf0bcda8-fb40-4404-abe6-434efa57889c" containerName="revoke-keys" Sep 06 06:13:22.486217 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.486160 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf0bcda8-fb40-4404-abe6-434efa57889c" containerName="revoke-keys" Sep 06 06:13:22.486267 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.486258 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf0bcda8-fb40-4404-abe6-434efa57889c" containerName="revoke-keys" Sep 06 06:13:22.486303 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.486269 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="29ab0249-1375-416c-906b-8ee4a6cd2dd9" containerName="istio-proxy" Sep 06 06:13:22.497008 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.496971 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj" Sep 06 06:13:22.499553 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.499332 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:13:22.499553 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.499479 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:13:22.499553 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.499515 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj"] Sep 06 06:13:22.544775 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.544743 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:13:22.545262 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.545128 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:13:22.545765 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.545743 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:13:22.554089 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.554060 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60a651b-7fe1-455b-a640-f1b1cdd92df0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj\" (UID: \"f60a651b-7fe1-455b-a640-f1b1cdd92df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj" Sep 06 06:13:22.654758 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.654725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60a651b-7fe1-455b-a640-f1b1cdd92df0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj\" (UID: \"f60a651b-7fe1-455b-a640-f1b1cdd92df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj" Sep 06 06:13:22.655380 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.655355 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60a651b-7fe1-455b-a640-f1b1cdd92df0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj\" (UID: \"f60a651b-7fe1-455b-a640-f1b1cdd92df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj" Sep 06 06:13:22.810301 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:22.810266 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj" Sep 06 06:13:23.080639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:23.080608 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj"] Sep 06 06:13:23.082114 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:13:23.082080 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60a651b_7fe1_455b_a640_f1b1cdd92df0.slice/crio-dbfe640de4f83535cf4c01f290fd1c6b811cf873f55f711ecc0022a368fa823d WatchSource:0}: Error finding container dbfe640de4f83535cf4c01f290fd1c6b811cf873f55f711ecc0022a368fa823d: Status 404 returned error can't find the container with id dbfe640de4f83535cf4c01f290fd1c6b811cf873f55f711ecc0022a368fa823d Sep 06 06:13:23.524470 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:23.524436 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj" event={"ID":"f60a651b-7fe1-455b-a640-f1b1cdd92df0","Type":"ContainerStarted","Data":"89c64ed0b8f7004278e804664a9b120cd4a7142d80cd9c424ebb71b0a76a7b2a"} Sep 06 06:13:23.524665 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:23.524498 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj" event={"ID":"f60a651b-7fe1-455b-a640-f1b1cdd92df0","Type":"ContainerStarted","Data":"dbfe640de4f83535cf4c01f290fd1c6b811cf873f55f711ecc0022a368fa823d"} Sep 06 06:13:23.538965 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:23.538918 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj" podStartSLOduration=1.5388815710000001 podStartE2EDuration="1.538881571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:13:23.537410704 +0000 UTC m=+1168.935741187" watchObservedRunningTime="2026-09-06 06:13:23.538881571 +0000 UTC m=+1168.937212038" Sep 06 06:13:24.532494 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:24.532458 2578 generic.go:356] "Generic (PLEG): container finished" podID="f60a651b-7fe1-455b-a640-f1b1cdd92df0" containerID="89c64ed0b8f7004278e804664a9b120cd4a7142d80cd9c424ebb71b0a76a7b2a" exitCode=0 Sep 06 06:13:24.532867 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:24.532549 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj" event={"ID":"f60a651b-7fe1-455b-a640-f1b1cdd92df0","Type":"ContainerDied","Data":"89c64ed0b8f7004278e804664a9b120cd4a7142d80cd9c424ebb71b0a76a7b2a"} Sep 06 06:13:25.678847 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:25.678826 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj" Sep 06 06:13:25.783959 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:25.783881 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60a651b-7fe1-455b-a640-f1b1cdd92df0-maas-api-service-ca\") pod \"f60a651b-7fe1-455b-a640-f1b1cdd92df0\" (UID: \"f60a651b-7fe1-455b-a640-f1b1cdd92df0\") " Sep 06 06:13:25.784244 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:25.784222 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f60a651b-7fe1-455b-a640-f1b1cdd92df0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f60a651b-7fe1-455b-a640-f1b1cdd92df0" (UID: "f60a651b-7fe1-455b-a640-f1b1cdd92df0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:13:25.885353 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:25.885256 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60a651b-7fe1-455b-a640-f1b1cdd92df0-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:26.544030 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:26.544001 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj" Sep 06 06:13:26.544183 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:26.544004 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj" event={"ID":"f60a651b-7fe1-455b-a640-f1b1cdd92df0","Type":"ContainerDied","Data":"dbfe640de4f83535cf4c01f290fd1c6b811cf873f55f711ecc0022a368fa823d"} Sep 06 06:13:26.544183 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:26.544119 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbfe640de4f83535cf4c01f290fd1c6b811cf873f55f711ecc0022a368fa823d" Sep 06 06:13:28.658988 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:28.658944 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54"] Sep 06 06:13:28.661046 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:28.661019 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-e75768-zsf54"] Sep 06 06:13:29.148472 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:29.148431 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf0bcda8-fb40-4404-abe6-434efa57889c" path="/var/lib/kubelet/pods/bf0bcda8-fb40-4404-abe6-434efa57889c/volumes" Sep 06 06:13:29.870485 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:29.870447 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj"] Sep 06 06:13:29.873166 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:29.873141 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f260fc83-2mbfj"] Sep 06 06:13:29.999730 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:29.999699 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq"] Sep 06 06:13:29.999951 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:29.999925 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" podUID="4f02530e-a95c-413a-a77a-11760ac19203" containerName="istio-proxy" containerID="cri-o://c7d9c6e58c1c50ab08835268cbe1616d6bec99458462952676e662e6dcfd8a98" gracePeriod=30 Sep 06 06:13:30.313945 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.313876 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" podUID="7397862c-8980-4407-a081-5d8e56326103" containerName="main" containerID="cri-o://aed6affe9678a6ee77f9f465c0f3f19a5d34b059fefa3e9fc2c4beac0bdc8d01" gracePeriod=45 Sep 06 06:13:30.568607 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.568541 2578 generic.go:356] "Generic (PLEG): container finished" podID="7397862c-8980-4407-a081-5d8e56326103" containerID="aed6affe9678a6ee77f9f465c0f3f19a5d34b059fefa3e9fc2c4beac0bdc8d01" exitCode=0 Sep 06 06:13:30.568607 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.568566 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" event={"ID":"7397862c-8980-4407-a081-5d8e56326103","Type":"ContainerDied","Data":"aed6affe9678a6ee77f9f465c0f3f19a5d34b059fefa3e9fc2c4beac0bdc8d01"} Sep 06 06:13:30.568607 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.568609 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" event={"ID":"7397862c-8980-4407-a081-5d8e56326103","Type":"ContainerDied","Data":"fffc1d2ec816ab0e830b47ad28af0be1f75c3910ab2ba9895b0ffe5fe45a7774"} Sep 06 06:13:30.568802 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.568625 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fffc1d2ec816ab0e830b47ad28af0be1f75c3910ab2ba9895b0ffe5fe45a7774" Sep 06 06:13:30.573599 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.573580 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:13:30.633278 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.633245 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-dshm\") pod \"7397862c-8980-4407-a081-5d8e56326103\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " Sep 06 06:13:30.633437 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.633343 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-home\") pod \"7397862c-8980-4407-a081-5d8e56326103\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " Sep 06 06:13:30.633437 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.633366 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7397862c-8980-4407-a081-5d8e56326103-tls-certs\") pod \"7397862c-8980-4407-a081-5d8e56326103\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " Sep 06 06:13:30.633437 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.633387 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvlf9\" (UniqueName: \"kubernetes.io/projected/7397862c-8980-4407-a081-5d8e56326103-kube-api-access-jvlf9\") pod \"7397862c-8980-4407-a081-5d8e56326103\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " Sep 06 06:13:30.633617 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.633450 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-tmp-dir\") pod \"7397862c-8980-4407-a081-5d8e56326103\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " Sep 06 06:13:30.633617 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.633509 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-model-cache\") pod \"7397862c-8980-4407-a081-5d8e56326103\" (UID: \"7397862c-8980-4407-a081-5d8e56326103\") " Sep 06 06:13:30.633728 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.633636 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-home" (OuterVolumeSpecName: "home") pod "7397862c-8980-4407-a081-5d8e56326103" (UID: "7397862c-8980-4407-a081-5d8e56326103"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:30.633818 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.633790 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7397862c-8980-4407-a081-5d8e56326103" (UID: "7397862c-8980-4407-a081-5d8e56326103"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:30.633889 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.633803 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-model-cache" (OuterVolumeSpecName: "model-cache") pod "7397862c-8980-4407-a081-5d8e56326103" (UID: "7397862c-8980-4407-a081-5d8e56326103"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:30.634186 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.634124 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-home\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:30.634186 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.634150 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-tmp-dir\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:30.634186 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.634164 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-model-cache\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:30.635734 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.635707 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-dshm" (OuterVolumeSpecName: "dshm") pod "7397862c-8980-4407-a081-5d8e56326103" (UID: "7397862c-8980-4407-a081-5d8e56326103"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:30.635980 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.635953 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7397862c-8980-4407-a081-5d8e56326103-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7397862c-8980-4407-a081-5d8e56326103" (UID: "7397862c-8980-4407-a081-5d8e56326103"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:13:30.636075 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.635975 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7397862c-8980-4407-a081-5d8e56326103-kube-api-access-jvlf9" (OuterVolumeSpecName: "kube-api-access-jvlf9") pod "7397862c-8980-4407-a081-5d8e56326103" (UID: "7397862c-8980-4407-a081-5d8e56326103"). InnerVolumeSpecName "kube-api-access-jvlf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:13:30.735375 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.735342 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7397862c-8980-4407-a081-5d8e56326103-tls-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:30.735375 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.735369 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvlf9\" (UniqueName: \"kubernetes.io/projected/7397862c-8980-4407-a081-5d8e56326103-kube-api-access-jvlf9\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:30.735375 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:30.735377 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7397862c-8980-4407-a081-5d8e56326103-dshm\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:31.145353 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:31.145321 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f60a651b-7fe1-455b-a640-f1b1cdd92df0" path="/var/lib/kubelet/pods/f60a651b-7fe1-455b-a640-f1b1cdd92df0/volumes" Sep 06 06:13:31.573966 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:31.573930 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn" Sep 06 06:13:31.589893 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:31.589864 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn"] Sep 06 06:13:31.597750 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:31.597711 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-d5939f9b-kserve-7bf57d6d68-8wmbn"] Sep 06 06:13:33.145554 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:33.145518 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7397862c-8980-4407-a081-5d8e56326103" path="/var/lib/kubelet/pods/7397862c-8980-4407-a081-5d8e56326103/volumes" Sep 06 06:13:34.475516 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:34.475474 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:13:35.023066 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.023034 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs"] Sep 06 06:13:35.023303 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.023272 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" podUID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerName="istio-proxy" containerID="cri-o://e16680b378cbb785588bf47b4bb679bed88d7889a220ec752e5f16c332c47172" gracePeriod=30 Sep 06 06:13:35.275075 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.275043 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:13:35.379477 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.379449 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-workload-certs\") pod \"4f02530e-a95c-413a-a77a-11760ac19203\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " Sep 06 06:13:35.379663 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.379490 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f02530e-a95c-413a-a77a-11760ac19203-istio-token\") pod \"4f02530e-a95c-413a-a77a-11760ac19203\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " Sep 06 06:13:35.379663 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.379514 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-workload-socket\") pod \"4f02530e-a95c-413a-a77a-11760ac19203\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " Sep 06 06:13:35.379663 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.379541 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-credential-socket\") pod \"4f02530e-a95c-413a-a77a-11760ac19203\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " Sep 06 06:13:35.379663 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.379588 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-istio-envoy\") pod \"4f02530e-a95c-413a-a77a-11760ac19203\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " Sep 06 06:13:35.379663 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.379611 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-istio-data\") pod \"4f02530e-a95c-413a-a77a-11760ac19203\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " Sep 06 06:13:35.379663 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.379661 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npjw6\" (UniqueName: \"kubernetes.io/projected/4f02530e-a95c-413a-a77a-11760ac19203-kube-api-access-npjw6\") pod \"4f02530e-a95c-413a-a77a-11760ac19203\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " Sep 06 06:13:35.380008 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.379746 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f02530e-a95c-413a-a77a-11760ac19203-istio-podinfo\") pod \"4f02530e-a95c-413a-a77a-11760ac19203\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " Sep 06 06:13:35.380008 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.379770 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4f02530e-a95c-413a-a77a-11760ac19203" (UID: "4f02530e-a95c-413a-a77a-11760ac19203"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:35.380008 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.379782 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f02530e-a95c-413a-a77a-11760ac19203-istiod-ca-cert\") pod \"4f02530e-a95c-413a-a77a-11760ac19203\" (UID: \"4f02530e-a95c-413a-a77a-11760ac19203\") " Sep 06 06:13:35.380008 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.379812 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4f02530e-a95c-413a-a77a-11760ac19203" (UID: "4f02530e-a95c-413a-a77a-11760ac19203"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:35.380239 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.380143 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f02530e-a95c-413a-a77a-11760ac19203-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4f02530e-a95c-413a-a77a-11760ac19203" (UID: "4f02530e-a95c-413a-a77a-11760ac19203"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:13:35.380239 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.380168 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4f02530e-a95c-413a-a77a-11760ac19203" (UID: "4f02530e-a95c-413a-a77a-11760ac19203"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:35.380344 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.380312 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:35.380344 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.380332 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:35.380446 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.380347 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f02530e-a95c-413a-a77a-11760ac19203-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:35.380446 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.380398 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:35.380535 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.380510 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-istio-data" (OuterVolumeSpecName: "istio-data") pod "4f02530e-a95c-413a-a77a-11760ac19203" (UID: "4f02530e-a95c-413a-a77a-11760ac19203"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:35.382586 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.382547 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f02530e-a95c-413a-a77a-11760ac19203-istio-token" (OuterVolumeSpecName: "istio-token") pod "4f02530e-a95c-413a-a77a-11760ac19203" (UID: "4f02530e-a95c-413a-a77a-11760ac19203"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:13:35.382711 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.382608 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f02530e-a95c-413a-a77a-11760ac19203-kube-api-access-npjw6" (OuterVolumeSpecName: "kube-api-access-npjw6") pod "4f02530e-a95c-413a-a77a-11760ac19203" (UID: "4f02530e-a95c-413a-a77a-11760ac19203"). InnerVolumeSpecName "kube-api-access-npjw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:13:35.382958 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.382931 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f02530e-a95c-413a-a77a-11760ac19203-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4f02530e-a95c-413a-a77a-11760ac19203" (UID: "4f02530e-a95c-413a-a77a-11760ac19203"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:13:35.382958 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.382937 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4f02530e-a95c-413a-a77a-11760ac19203" (UID: "4f02530e-a95c-413a-a77a-11760ac19203"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:35.481970 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.481894 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:35.482328 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.481977 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npjw6\" (UniqueName: \"kubernetes.io/projected/4f02530e-a95c-413a-a77a-11760ac19203-kube-api-access-npjw6\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:35.482328 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.481995 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f02530e-a95c-413a-a77a-11760ac19203-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:35.482328 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.482010 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f02530e-a95c-413a-a77a-11760ac19203-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:35.482328 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.482023 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f02530e-a95c-413a-a77a-11760ac19203-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:35.601581 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.601470 2578 generic.go:356] "Generic (PLEG): container finished" podID="4f02530e-a95c-413a-a77a-11760ac19203" containerID="c7d9c6e58c1c50ab08835268cbe1616d6bec99458462952676e662e6dcfd8a98" exitCode=0 Sep 06 06:13:35.601581 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.601534 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" Sep 06 06:13:35.601581 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.601557 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" event={"ID":"4f02530e-a95c-413a-a77a-11760ac19203","Type":"ContainerDied","Data":"c7d9c6e58c1c50ab08835268cbe1616d6bec99458462952676e662e6dcfd8a98"} Sep 06 06:13:35.601889 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.601615 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq" event={"ID":"4f02530e-a95c-413a-a77a-11760ac19203","Type":"ContainerDied","Data":"edebec35c57407efb9553e0833971821044db4f144781d074a0a953059df91c3"} Sep 06 06:13:35.601889 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.601641 2578 scope.go:117] "RemoveContainer" containerID="c7d9c6e58c1c50ab08835268cbe1616d6bec99458462952676e662e6dcfd8a98" Sep 06 06:13:35.612052 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.612028 2578 scope.go:117] "RemoveContainer" containerID="2ef333a3f526265f2c863a8aecb7721c2e4d6ec1f7f676518334def20a0d79c2" Sep 06 06:13:35.621610 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.621589 2578 scope.go:117] "RemoveContainer" containerID="c7d9c6e58c1c50ab08835268cbe1616d6bec99458462952676e662e6dcfd8a98" Sep 06 06:13:35.621991 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:13:35.621893 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d9c6e58c1c50ab08835268cbe1616d6bec99458462952676e662e6dcfd8a98\": container with ID starting with c7d9c6e58c1c50ab08835268cbe1616d6bec99458462952676e662e6dcfd8a98 not found: ID does not exist" containerID="c7d9c6e58c1c50ab08835268cbe1616d6bec99458462952676e662e6dcfd8a98" Sep 06 06:13:35.622039 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.622008 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d9c6e58c1c50ab08835268cbe1616d6bec99458462952676e662e6dcfd8a98"} err="failed to get container status \"c7d9c6e58c1c50ab08835268cbe1616d6bec99458462952676e662e6dcfd8a98\": rpc error: code = NotFound desc = could not find container \"c7d9c6e58c1c50ab08835268cbe1616d6bec99458462952676e662e6dcfd8a98\": container with ID starting with c7d9c6e58c1c50ab08835268cbe1616d6bec99458462952676e662e6dcfd8a98 not found: ID does not exist" Sep 06 06:13:35.622039 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.622033 2578 scope.go:117] "RemoveContainer" containerID="2ef333a3f526265f2c863a8aecb7721c2e4d6ec1f7f676518334def20a0d79c2" Sep 06 06:13:35.622310 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:13:35.622283 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef333a3f526265f2c863a8aecb7721c2e4d6ec1f7f676518334def20a0d79c2\": container with ID starting with 2ef333a3f526265f2c863a8aecb7721c2e4d6ec1f7f676518334def20a0d79c2 not found: ID does not exist" containerID="2ef333a3f526265f2c863a8aecb7721c2e4d6ec1f7f676518334def20a0d79c2" Sep 06 06:13:35.622358 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.622316 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef333a3f526265f2c863a8aecb7721c2e4d6ec1f7f676518334def20a0d79c2"} err="failed to get container status \"2ef333a3f526265f2c863a8aecb7721c2e4d6ec1f7f676518334def20a0d79c2\": rpc error: code = NotFound desc = could not find container \"2ef333a3f526265f2c863a8aecb7721c2e4d6ec1f7f676518334def20a0d79c2\": container with ID starting with 2ef333a3f526265f2c863a8aecb7721c2e4d6ec1f7f676518334def20a0d79c2 not found: ID does not exist" Sep 06 06:13:35.627942 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.627889 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq"] Sep 06 06:13:35.633121 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.633095 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f260fc83-openshift-default-748b6897bd-dndzq"] Sep 06 06:13:35.726564 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.726519 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2"] Sep 06 06:13:35.727134 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.727114 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f60a651b-7fe1-455b-a640-f1b1cdd92df0" containerName="revoke-keys" Sep 06 06:13:35.727134 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.727134 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f60a651b-7fe1-455b-a640-f1b1cdd92df0" containerName="revoke-keys" Sep 06 06:13:35.727352 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.727153 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f02530e-a95c-413a-a77a-11760ac19203" containerName="istio-proxy" Sep 06 06:13:35.727352 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.727162 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f02530e-a95c-413a-a77a-11760ac19203" containerName="istio-proxy" Sep 06 06:13:35.727352 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.727180 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f02530e-a95c-413a-a77a-11760ac19203" containerName="istio-proxy" Sep 06 06:13:35.727352 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.727189 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f02530e-a95c-413a-a77a-11760ac19203" containerName="istio-proxy" Sep 06 06:13:35.727352 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.727219 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7397862c-8980-4407-a081-5d8e56326103" containerName="main" Sep 06 06:13:35.727352 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.727228 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7397862c-8980-4407-a081-5d8e56326103" containerName="main" Sep 06 06:13:35.727352 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.727321 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7397862c-8980-4407-a081-5d8e56326103" containerName="main" Sep 06 06:13:35.727352 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.727331 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f02530e-a95c-413a-a77a-11760ac19203" containerName="istio-proxy" Sep 06 06:13:35.727352 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.727345 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f60a651b-7fe1-455b-a640-f1b1cdd92df0" containerName="revoke-keys" Sep 06 06:13:35.732208 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.732183 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.734248 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.734225 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-c50b6f-openshift-default-dockercfg-7czm8\"" Sep 06 06:13:35.739319 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.739294 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2"] Sep 06 06:13:35.785683 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.785645 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-token\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.785882 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.785698 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-credential-socket\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.785882 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.785746 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-workload-certs\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.785882 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.785772 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-workload-socket\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.786068 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.785915 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-data\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.786068 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.785985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5faad5e5-5edc-4d42-bc3d-012a8834d072-istiod-ca-cert\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.786068 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.786037 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-podinfo\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.786183 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.786108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dszbd\" (UniqueName: \"kubernetes.io/projected/5faad5e5-5edc-4d42-bc3d-012a8834d072-kube-api-access-dszbd\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.786183 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.786162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-envoy\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.887276 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.887174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-data\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.887276 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.887245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5faad5e5-5edc-4d42-bc3d-012a8834d072-istiod-ca-cert\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.887490 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.887297 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-podinfo\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.887490 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.887446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dszbd\" (UniqueName: \"kubernetes.io/projected/5faad5e5-5edc-4d42-bc3d-012a8834d072-kube-api-access-dszbd\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.887606 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.887503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-envoy\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.887606 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.887585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-token\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.887712 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.887620 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-credential-socket\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.887712 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.887655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-workload-certs\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.887712 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.887674 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-data\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.887712 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.887685 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-workload-socket\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.888013 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.887990 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-workload-socket\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.897594 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.888550 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5faad5e5-5edc-4d42-bc3d-012a8834d072-istiod-ca-cert\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.897594 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.888768 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-credential-socket\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.899690 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.899657 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-workload-certs\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.899814 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.899719 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-podinfo\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.902411 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.902376 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-envoy\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.902508 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.902409 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-token\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:35.906238 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:35.906216 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dszbd\" (UniqueName: \"kubernetes.io/projected/5faad5e5-5edc-4d42-bc3d-012a8834d072-kube-api-access-dszbd\") pod \"e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:36.044272 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:36.044235 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:36.318946 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:36.318918 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2"] Sep 06 06:13:36.320849 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:13:36.320815 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5faad5e5_5edc_4d42_bc3d_012a8834d072.slice/crio-f8ea6e593867615d2ba59affcf607723fea5a9f3b6e8657300bdf45297ad47b2 WatchSource:0}: Error finding container f8ea6e593867615d2ba59affcf607723fea5a9f3b6e8657300bdf45297ad47b2: Status 404 returned error can't find the container with id f8ea6e593867615d2ba59affcf607723fea5a9f3b6e8657300bdf45297ad47b2 Sep 06 06:13:36.323140 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:36.323106 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:13:36.323247 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:36.323192 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:13:36.323247 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:36.323222 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:13:36.609454 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:36.609363 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" event={"ID":"5faad5e5-5edc-4d42-bc3d-012a8834d072","Type":"ContainerStarted","Data":"510efe6d9e4f1d632b679fbab078c8b0d11ce369e6c0dce750bd30b40e517f36"} Sep 06 06:13:36.609454 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:36.609417 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" event={"ID":"5faad5e5-5edc-4d42-bc3d-012a8834d072","Type":"ContainerStarted","Data":"f8ea6e593867615d2ba59affcf607723fea5a9f3b6e8657300bdf45297ad47b2"} Sep 06 06:13:36.637561 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:36.637505 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" podStartSLOduration=1.637486703 podStartE2EDuration="1.637486703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:13:36.633663131 +0000 UTC m=+1182.031993622" watchObservedRunningTime="2026-09-06 06:13:36.637486703 +0000 UTC m=+1182.035817162" Sep 06 06:13:37.044980 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:37.044945 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:37.046304 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:37.046276 2578 patch_prober.go:28] interesting pod/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 06 06:13:37.046432 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:37.046327 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 06 06:13:37.145478 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:37.145447 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f02530e-a95c-413a-a77a-11760ac19203" path="/var/lib/kubelet/pods/4f02530e-a95c-413a-a77a-11760ac19203/volumes" Sep 06 06:13:38.049338 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:38.049308 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:38.622170 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:38.622142 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:38.623021 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:38.623003 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:40.292736 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.292670 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:13:40.433220 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.433181 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p89j8\" (UniqueName: \"kubernetes.io/projected/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-kube-api-access-p89j8\") pod \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " Sep 06 06:13:40.433399 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.433235 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istiod-ca-cert\") pod \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " Sep 06 06:13:40.433399 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.433280 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-workload-certs\") pod \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " Sep 06 06:13:40.433399 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.433362 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-credential-socket\") pod \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " Sep 06 06:13:40.433543 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.433405 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-token\") pod \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " Sep 06 06:13:40.433543 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.433429 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-podinfo\") pod \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " Sep 06 06:13:40.433639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.433576 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-envoy\") pod \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " Sep 06 06:13:40.433639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.433603 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "97058a65-bc0c-44d0-86d7-1320b9d8a5e8" (UID: "97058a65-bc0c-44d0-86d7-1320b9d8a5e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:40.433745 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.433639 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-data\") pod \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " Sep 06 06:13:40.433745 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.433656 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "97058a65-bc0c-44d0-86d7-1320b9d8a5e8" (UID: "97058a65-bc0c-44d0-86d7-1320b9d8a5e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:13:40.433745 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.433661 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "97058a65-bc0c-44d0-86d7-1320b9d8a5e8" (UID: "97058a65-bc0c-44d0-86d7-1320b9d8a5e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:40.433745 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.433699 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-workload-socket\") pod \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\" (UID: \"97058a65-bc0c-44d0-86d7-1320b9d8a5e8\") " Sep 06 06:13:40.433972 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.433827 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-data" (OuterVolumeSpecName: "istio-data") pod "97058a65-bc0c-44d0-86d7-1320b9d8a5e8" (UID: "97058a65-bc0c-44d0-86d7-1320b9d8a5e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:40.434037 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.433979 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "97058a65-bc0c-44d0-86d7-1320b9d8a5e8" (UID: "97058a65-bc0c-44d0-86d7-1320b9d8a5e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:40.434404 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.434385 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:40.434483 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.434410 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:40.434483 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.434425 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:40.434483 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.434440 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:40.434483 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.434453 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:40.435783 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.435742 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-kube-api-access-p89j8" (OuterVolumeSpecName: "kube-api-access-p89j8") pod "97058a65-bc0c-44d0-86d7-1320b9d8a5e8" (UID: "97058a65-bc0c-44d0-86d7-1320b9d8a5e8"). InnerVolumeSpecName "kube-api-access-p89j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:13:40.435896 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.435849 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "97058a65-bc0c-44d0-86d7-1320b9d8a5e8" (UID: "97058a65-bc0c-44d0-86d7-1320b9d8a5e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:13:40.436229 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.436211 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-token" (OuterVolumeSpecName: "istio-token") pod "97058a65-bc0c-44d0-86d7-1320b9d8a5e8" (UID: "97058a65-bc0c-44d0-86d7-1320b9d8a5e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:13:40.436279 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.436209 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "97058a65-bc0c-44d0-86d7-1320b9d8a5e8" (UID: "97058a65-bc0c-44d0-86d7-1320b9d8a5e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:13:40.535469 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.535427 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:40.535469 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.535460 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:40.535469 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.535473 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:40.535697 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.535484 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p89j8\" (UniqueName: \"kubernetes.io/projected/97058a65-bc0c-44d0-86d7-1320b9d8a5e8-kube-api-access-p89j8\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:13:40.634992 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.634887 2578 generic.go:356] "Generic (PLEG): container finished" podID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerID="e16680b378cbb785588bf47b4bb679bed88d7889a220ec752e5f16c332c47172" exitCode=0 Sep 06 06:13:40.634992 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.634950 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" Sep 06 06:13:40.634992 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.634980 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" event={"ID":"97058a65-bc0c-44d0-86d7-1320b9d8a5e8","Type":"ContainerDied","Data":"e16680b378cbb785588bf47b4bb679bed88d7889a220ec752e5f16c332c47172"} Sep 06 06:13:40.635269 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.635061 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs" event={"ID":"97058a65-bc0c-44d0-86d7-1320b9d8a5e8","Type":"ContainerDied","Data":"c797ee7bb811d49eeb4e7aa57130ed3afa028513a62a44f522287e3eeeed8035"} Sep 06 06:13:40.635269 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.635084 2578 scope.go:117] "RemoveContainer" containerID="e16680b378cbb785588bf47b4bb679bed88d7889a220ec752e5f16c332c47172" Sep 06 06:13:40.645579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.645560 2578 scope.go:117] "RemoveContainer" containerID="8d395ad83ce99da7e5af6d1e79e7e4354e0a833eceeacef77e0b1a986c0a6338" Sep 06 06:13:40.654140 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.654120 2578 scope.go:117] "RemoveContainer" containerID="e16680b378cbb785588bf47b4bb679bed88d7889a220ec752e5f16c332c47172" Sep 06 06:13:40.654453 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:13:40.654435 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16680b378cbb785588bf47b4bb679bed88d7889a220ec752e5f16c332c47172\": container with ID starting with e16680b378cbb785588bf47b4bb679bed88d7889a220ec752e5f16c332c47172 not found: ID does not exist" containerID="e16680b378cbb785588bf47b4bb679bed88d7889a220ec752e5f16c332c47172" Sep 06 06:13:40.654502 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.654465 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16680b378cbb785588bf47b4bb679bed88d7889a220ec752e5f16c332c47172"} err="failed to get container status \"e16680b378cbb785588bf47b4bb679bed88d7889a220ec752e5f16c332c47172\": rpc error: code = NotFound desc = could not find container \"e16680b378cbb785588bf47b4bb679bed88d7889a220ec752e5f16c332c47172\": container with ID starting with e16680b378cbb785588bf47b4bb679bed88d7889a220ec752e5f16c332c47172 not found: ID does not exist" Sep 06 06:13:40.654502 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.654483 2578 scope.go:117] "RemoveContainer" containerID="8d395ad83ce99da7e5af6d1e79e7e4354e0a833eceeacef77e0b1a986c0a6338" Sep 06 06:13:40.654748 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:13:40.654732 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d395ad83ce99da7e5af6d1e79e7e4354e0a833eceeacef77e0b1a986c0a6338\": container with ID starting with 8d395ad83ce99da7e5af6d1e79e7e4354e0a833eceeacef77e0b1a986c0a6338 not found: ID does not exist" containerID="8d395ad83ce99da7e5af6d1e79e7e4354e0a833eceeacef77e0b1a986c0a6338" Sep 06 06:13:40.654804 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.654754 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d395ad83ce99da7e5af6d1e79e7e4354e0a833eceeacef77e0b1a986c0a6338"} err="failed to get container status \"8d395ad83ce99da7e5af6d1e79e7e4354e0a833eceeacef77e0b1a986c0a6338\": rpc error: code = NotFound desc = could not find container \"8d395ad83ce99da7e5af6d1e79e7e4354e0a833eceeacef77e0b1a986c0a6338\": container with ID starting with 8d395ad83ce99da7e5af6d1e79e7e4354e0a833eceeacef77e0b1a986c0a6338 not found: ID does not exist" Sep 06 06:13:40.657885 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.657864 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs"] Sep 06 06:13:40.661975 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:40.661945 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-e75768-openshift-default-5ffcf9c7cd-pgscs"] Sep 06 06:13:41.147695 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:41.147661 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" path="/var/lib/kubelet/pods/97058a65-bc0c-44d0-86d7-1320b9d8a5e8/volumes" Sep 06 06:13:51.679498 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:51.679461 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:13:53.623648 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:53.623609 2578 patch_prober.go:28] interesting pod/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 06 06:13:53.624111 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:53.623672 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 06 06:13:53.725116 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:53.725080 2578 generic.go:356] "Generic (PLEG): container finished" podID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerID="510efe6d9e4f1d632b679fbab078c8b0d11ce369e6c0dce750bd30b40e517f36" exitCode=0 Sep 06 06:13:53.725287 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:53.725150 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" event={"ID":"5faad5e5-5edc-4d42-bc3d-012a8834d072","Type":"ContainerDied","Data":"510efe6d9e4f1d632b679fbab078c8b0d11ce369e6c0dce750bd30b40e517f36"} Sep 06 06:13:53.725589 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:53.725563 2578 scope.go:117] "RemoveContainer" containerID="510efe6d9e4f1d632b679fbab078c8b0d11ce369e6c0dce750bd30b40e517f36" Sep 06 06:13:53.726730 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:53.726704 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:13:53.726780 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:53.726769 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:13:53.726825 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:53.726801 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:13:54.733533 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:54.733503 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" event={"ID":"5faad5e5-5edc-4d42-bc3d-012a8834d072","Type":"ContainerStarted","Data":"9e1587fea7826bc5d5e640ee47a0bbf086b2a870179c5d58d60b0455b43d51d0"} Sep 06 06:13:55.044775 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:55.044739 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:13:55.044973 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:55.044943 2578 patch_prober.go:28] interesting pod/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 06 06:13:55.045030 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:55.045016 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 06 06:13:55.125167 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:55.125141 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 06:13:55.127682 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:55.127660 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 06:13:56.045612 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:56.045576 2578 patch_prober.go:28] interesting pod/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 06 06:13:56.046056 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:56.045637 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 06 06:13:57.045507 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:57.045462 2578 patch_prober.go:28] interesting pod/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 06 06:13:57.045690 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:57.045524 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 06 06:13:58.044804 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:58.044759 2578 patch_prober.go:28] interesting pod/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 06 06:13:58.045036 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:58.044815 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 06 06:13:59.044949 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:59.044884 2578 patch_prober.go:28] interesting pod/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 06 06:13:59.045420 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:13:59.044977 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 06 06:14:00.066950 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:00.066858 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:14:00.067511 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:00.067485 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:14:00.068365 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:00.068336 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:14:03.279316 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:03.279276 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:14:15.985189 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:15.985152 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:14:27.304460 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:27.304432 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s"] Sep 06 06:14:27.304939 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:27.304818 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerName="istio-proxy" Sep 06 06:14:27.304939 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:27.304828 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerName="istio-proxy" Sep 06 06:14:27.305020 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:27.304954 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f02530e-a95c-413a-a77a-11760ac19203" containerName="istio-proxy" Sep 06 06:14:27.305020 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:27.304965 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerName="istio-proxy" Sep 06 06:14:27.308014 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:27.307997 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s" Sep 06 06:14:27.310044 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:27.310017 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:14:27.310149 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:27.310065 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:14:27.314357 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:27.314338 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s"] Sep 06 06:14:27.397297 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:27.397267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4c295d-4b1c-42fa-a064-89ae81da4145-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s\" (UID: \"af4c295d-4b1c-42fa-a064-89ae81da4145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s" Sep 06 06:14:27.498674 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:27.498642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4c295d-4b1c-42fa-a064-89ae81da4145-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s\" (UID: \"af4c295d-4b1c-42fa-a064-89ae81da4145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s" Sep 06 06:14:27.499304 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:27.499285 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4c295d-4b1c-42fa-a064-89ae81da4145-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s\" (UID: \"af4c295d-4b1c-42fa-a064-89ae81da4145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s" Sep 06 06:14:27.620897 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:27.620809 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s" Sep 06 06:14:27.890361 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:27.890335 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s"] Sep 06 06:14:27.891967 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:14:27.891942 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4c295d_4b1c_42fa_a064_89ae81da4145.slice/crio-db86c8fd9a3b6614848789519c1110024d7f80404b5570c0375389e041586f98 WatchSource:0}: Error finding container db86c8fd9a3b6614848789519c1110024d7f80404b5570c0375389e041586f98: Status 404 returned error can't find the container with id db86c8fd9a3b6614848789519c1110024d7f80404b5570c0375389e041586f98 Sep 06 06:14:27.922091 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:27.922061 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s" event={"ID":"af4c295d-4b1c-42fa-a064-89ae81da4145","Type":"ContainerStarted","Data":"db86c8fd9a3b6614848789519c1110024d7f80404b5570c0375389e041586f98"} Sep 06 06:14:28.929366 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:28.929284 2578 generic.go:356] "Generic (PLEG): container finished" podID="af4c295d-4b1c-42fa-a064-89ae81da4145" containerID="d249e990103876aac41ca6ad15f76df626792a21de0264875c27ab7e79c8fb1a" exitCode=0 Sep 06 06:14:28.929699 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:28.929372 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s" event={"ID":"af4c295d-4b1c-42fa-a064-89ae81da4145","Type":"ContainerDied","Data":"d249e990103876aac41ca6ad15f76df626792a21de0264875c27ab7e79c8fb1a"} Sep 06 06:14:30.070846 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:30.070819 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s" Sep 06 06:14:30.224982 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:30.224878 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4c295d-4b1c-42fa-a064-89ae81da4145-maas-api-service-ca\") pod \"af4c295d-4b1c-42fa-a064-89ae81da4145\" (UID: \"af4c295d-4b1c-42fa-a064-89ae81da4145\") " Sep 06 06:14:30.225301 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:30.225276 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4c295d-4b1c-42fa-a064-89ae81da4145-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af4c295d-4b1c-42fa-a064-89ae81da4145" (UID: "af4c295d-4b1c-42fa-a064-89ae81da4145"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:14:30.325827 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:30.325799 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4c295d-4b1c-42fa-a064-89ae81da4145-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:14:30.941094 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:30.941061 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s" event={"ID":"af4c295d-4b1c-42fa-a064-89ae81da4145","Type":"ContainerDied","Data":"db86c8fd9a3b6614848789519c1110024d7f80404b5570c0375389e041586f98"} Sep 06 06:14:30.941094 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:30.941079 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s" Sep 06 06:14:30.941305 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:30.941107 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db86c8fd9a3b6614848789519c1110024d7f80404b5570c0375389e041586f98" Sep 06 06:14:31.076217 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:31.076182 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:14:37.942020 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:37.941989 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s"] Sep 06 06:14:37.943890 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:37.943864 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c50b6f-kvr7s"] Sep 06 06:14:39.151920 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:39.151870 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af4c295d-4b1c-42fa-a064-89ae81da4145" path="/var/lib/kubelet/pods/af4c295d-4b1c-42fa-a064-89ae81da4145/volumes" Sep 06 06:14:44.113024 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.112990 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2"] Sep 06 06:14:44.113418 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.113259 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerName="istio-proxy" containerID="cri-o://9e1587fea7826bc5d5e640ee47a0bbf086b2a870179c5d58d60b0455b43d51d0" gracePeriod=30 Sep 06 06:14:44.437270 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.437184 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:14:44.862850 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.862807 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf"] Sep 06 06:14:44.863242 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.863223 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerName="istio-proxy" Sep 06 06:14:44.863242 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.863239 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerName="istio-proxy" Sep 06 06:14:44.863391 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.863251 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4c295d-4b1c-42fa-a064-89ae81da4145" containerName="revoke-keys" Sep 06 06:14:44.863391 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.863257 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4c295d-4b1c-42fa-a064-89ae81da4145" containerName="revoke-keys" Sep 06 06:14:44.863391 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.863329 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="97058a65-bc0c-44d0-86d7-1320b9d8a5e8" containerName="istio-proxy" Sep 06 06:14:44.863391 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.863339 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="af4c295d-4b1c-42fa-a064-89ae81da4145" containerName="revoke-keys" Sep 06 06:14:44.867594 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.867572 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:44.869640 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.869619 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-837ae9-openshift-default-dockercfg-5mznq\"" Sep 06 06:14:44.876221 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.876198 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf"] Sep 06 06:14:44.962723 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.962621 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-workload-socket\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:44.962932 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.962747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-envoy\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:44.962932 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.962774 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-credential-socket\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:44.962932 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.962838 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j7wh\" (UniqueName: \"kubernetes.io/projected/1895bcbc-ea91-4458-b92c-de76f65b65ce-kube-api-access-8j7wh\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:44.962932 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.962874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-workload-certs\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:44.963180 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.962940 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-token\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:44.963180 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.963028 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-podinfo\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:44.963180 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.963054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1895bcbc-ea91-4458-b92c-de76f65b65ce-istiod-ca-cert\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:44.963180 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:44.963089 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-data\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.063784 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.063744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-podinfo\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.063784 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.063781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1895bcbc-ea91-4458-b92c-de76f65b65ce-istiod-ca-cert\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.064081 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.063804 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-data\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.064081 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.063835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-workload-socket\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.064081 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.063853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-envoy\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.064081 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.063868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-credential-socket\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.064081 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.063947 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j7wh\" (UniqueName: \"kubernetes.io/projected/1895bcbc-ea91-4458-b92c-de76f65b65ce-kube-api-access-8j7wh\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.064081 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.063980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-workload-certs\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.064081 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.064010 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-token\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.064452 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.064303 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-data\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.064452 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.064328 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-workload-socket\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.064452 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.064441 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-workload-certs\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.064609 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.064583 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1895bcbc-ea91-4458-b92c-de76f65b65ce-istiod-ca-cert\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.064931 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.064647 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-credential-socket\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.066449 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.066420 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-envoy\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.066680 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.066659 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-podinfo\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.069546 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.069523 2578 patch_prober.go:28] interesting pod/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 06 06:14:45.069642 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.069583 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 06 06:14:45.074011 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.073993 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-token\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.074129 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.074111 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j7wh\" (UniqueName: \"kubernetes.io/projected/1895bcbc-ea91-4458-b92c-de76f65b65ce-kube-api-access-8j7wh\") pod \"e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.178818 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.178718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:45.458124 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.458040 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf"] Sep 06 06:14:45.463334 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:14:45.463308 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1895bcbc_ea91_4458_b92c_de76f65b65ce.slice/crio-377124b201d12242c79d65903a5d03d4dde6d3f4d4ef22332db03e3ef669dc92 WatchSource:0}: Error finding container 377124b201d12242c79d65903a5d03d4dde6d3f4d4ef22332db03e3ef669dc92: Status 404 returned error can't find the container with id 377124b201d12242c79d65903a5d03d4dde6d3f4d4ef22332db03e3ef669dc92 Sep 06 06:14:45.466179 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.466142 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:14:45.466268 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.466230 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:14:45.466318 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:45.466278 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:14:46.033072 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:46.032977 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" event={"ID":"1895bcbc-ea91-4458-b92c-de76f65b65ce","Type":"ContainerStarted","Data":"a951f7ef8c455860d93326ce41b5a878331a9e16b3dcc7a1eac492e73c67a3e7"} Sep 06 06:14:46.033072 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:46.033040 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" event={"ID":"1895bcbc-ea91-4458-b92c-de76f65b65ce","Type":"ContainerStarted","Data":"377124b201d12242c79d65903a5d03d4dde6d3f4d4ef22332db03e3ef669dc92"} Sep 06 06:14:46.050287 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:46.050234 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" podStartSLOduration=2.050218385 podStartE2EDuration="2.050218385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:14:46.049070285 +0000 UTC m=+1251.447400765" watchObservedRunningTime="2026-09-06 06:14:46.050218385 +0000 UTC m=+1251.448548854" Sep 06 06:14:46.179240 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:46.179200 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:46.180665 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:46.180643 2578 patch_prober.go:28] interesting pod/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 06 06:14:46.180720 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:46.180696 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" podUID="1895bcbc-ea91-4458-b92c-de76f65b65ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 06 06:14:47.183125 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:47.183093 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:48.043893 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:48.043864 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:48.044752 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:48.044733 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:14:49.368238 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.368212 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:14:49.410534 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.410433 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-data\") pod \"5faad5e5-5edc-4d42-bc3d-012a8834d072\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " Sep 06 06:14:49.410534 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.410467 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5faad5e5-5edc-4d42-bc3d-012a8834d072-istiod-ca-cert\") pod \"5faad5e5-5edc-4d42-bc3d-012a8834d072\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " Sep 06 06:14:49.410534 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.410524 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-credential-socket\") pod \"5faad5e5-5edc-4d42-bc3d-012a8834d072\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " Sep 06 06:14:49.410828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.410557 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dszbd\" (UniqueName: \"kubernetes.io/projected/5faad5e5-5edc-4d42-bc3d-012a8834d072-kube-api-access-dszbd\") pod \"5faad5e5-5edc-4d42-bc3d-012a8834d072\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " Sep 06 06:14:49.410828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.410605 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-token\") pod \"5faad5e5-5edc-4d42-bc3d-012a8834d072\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " Sep 06 06:14:49.410828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.410646 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-workload-socket\") pod \"5faad5e5-5edc-4d42-bc3d-012a8834d072\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " Sep 06 06:14:49.410828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.410682 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-envoy\") pod \"5faad5e5-5edc-4d42-bc3d-012a8834d072\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " Sep 06 06:14:49.410828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.410707 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-workload-certs\") pod \"5faad5e5-5edc-4d42-bc3d-012a8834d072\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " Sep 06 06:14:49.410828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.410727 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-data" (OuterVolumeSpecName: "istio-data") pod "5faad5e5-5edc-4d42-bc3d-012a8834d072" (UID: "5faad5e5-5edc-4d42-bc3d-012a8834d072"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:14:49.410828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.410743 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-podinfo\") pod \"5faad5e5-5edc-4d42-bc3d-012a8834d072\" (UID: \"5faad5e5-5edc-4d42-bc3d-012a8834d072\") " Sep 06 06:14:49.411245 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.410874 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5faad5e5-5edc-4d42-bc3d-012a8834d072" (UID: "5faad5e5-5edc-4d42-bc3d-012a8834d072"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:14:49.411245 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.410938 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5faad5e5-5edc-4d42-bc3d-012a8834d072-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5faad5e5-5edc-4d42-bc3d-012a8834d072" (UID: "5faad5e5-5edc-4d42-bc3d-012a8834d072"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:14:49.411245 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.411046 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5faad5e5-5edc-4d42-bc3d-012a8834d072" (UID: "5faad5e5-5edc-4d42-bc3d-012a8834d072"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:14:49.411245 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.411240 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:14:49.411470 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.411262 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5faad5e5-5edc-4d42-bc3d-012a8834d072-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:14:49.411470 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.411277 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:14:49.411470 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.411292 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:14:49.411470 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.411240 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5faad5e5-5edc-4d42-bc3d-012a8834d072" (UID: "5faad5e5-5edc-4d42-bc3d-012a8834d072"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:14:49.413059 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.413034 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5faad5e5-5edc-4d42-bc3d-012a8834d072-kube-api-access-dszbd" (OuterVolumeSpecName: "kube-api-access-dszbd") pod "5faad5e5-5edc-4d42-bc3d-012a8834d072" (UID: "5faad5e5-5edc-4d42-bc3d-012a8834d072"). InnerVolumeSpecName "kube-api-access-dszbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:14:49.413383 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.413352 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5faad5e5-5edc-4d42-bc3d-012a8834d072" (UID: "5faad5e5-5edc-4d42-bc3d-012a8834d072"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:14:49.413525 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.413506 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5faad5e5-5edc-4d42-bc3d-012a8834d072" (UID: "5faad5e5-5edc-4d42-bc3d-012a8834d072"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:14:49.413600 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.413523 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-token" (OuterVolumeSpecName: "istio-token") pod "5faad5e5-5edc-4d42-bc3d-012a8834d072" (UID: "5faad5e5-5edc-4d42-bc3d-012a8834d072"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:14:49.512400 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.512365 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dszbd\" (UniqueName: \"kubernetes.io/projected/5faad5e5-5edc-4d42-bc3d-012a8834d072-kube-api-access-dszbd\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:14:49.512400 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.512394 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:14:49.512400 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.512403 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:14:49.512400 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.512411 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5faad5e5-5edc-4d42-bc3d-012a8834d072-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:14:49.512664 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:49.512418 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5faad5e5-5edc-4d42-bc3d-012a8834d072-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:14:50.057351 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:50.057317 2578 generic.go:356] "Generic (PLEG): container finished" podID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerID="9e1587fea7826bc5d5e640ee47a0bbf086b2a870179c5d58d60b0455b43d51d0" exitCode=0 Sep 06 06:14:50.057564 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:50.057364 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" Sep 06 06:14:50.057564 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:50.057400 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" event={"ID":"5faad5e5-5edc-4d42-bc3d-012a8834d072","Type":"ContainerDied","Data":"9e1587fea7826bc5d5e640ee47a0bbf086b2a870179c5d58d60b0455b43d51d0"} Sep 06 06:14:50.057564 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:50.057469 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2" event={"ID":"5faad5e5-5edc-4d42-bc3d-012a8834d072","Type":"ContainerDied","Data":"f8ea6e593867615d2ba59affcf607723fea5a9f3b6e8657300bdf45297ad47b2"} Sep 06 06:14:50.057564 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:50.057502 2578 scope.go:117] "RemoveContainer" containerID="9e1587fea7826bc5d5e640ee47a0bbf086b2a870179c5d58d60b0455b43d51d0" Sep 06 06:14:50.069532 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:50.069511 2578 scope.go:117] "RemoveContainer" containerID="510efe6d9e4f1d632b679fbab078c8b0d11ce369e6c0dce750bd30b40e517f36" Sep 06 06:14:50.078151 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:50.078132 2578 scope.go:117] "RemoveContainer" containerID="9e1587fea7826bc5d5e640ee47a0bbf086b2a870179c5d58d60b0455b43d51d0" Sep 06 06:14:50.078432 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:14:50.078411 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e1587fea7826bc5d5e640ee47a0bbf086b2a870179c5d58d60b0455b43d51d0\": container with ID starting with 9e1587fea7826bc5d5e640ee47a0bbf086b2a870179c5d58d60b0455b43d51d0 not found: ID does not exist" containerID="9e1587fea7826bc5d5e640ee47a0bbf086b2a870179c5d58d60b0455b43d51d0" Sep 06 06:14:50.078481 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:50.078444 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e1587fea7826bc5d5e640ee47a0bbf086b2a870179c5d58d60b0455b43d51d0"} err="failed to get container status \"9e1587fea7826bc5d5e640ee47a0bbf086b2a870179c5d58d60b0455b43d51d0\": rpc error: code = NotFound desc = could not find container \"9e1587fea7826bc5d5e640ee47a0bbf086b2a870179c5d58d60b0455b43d51d0\": container with ID starting with 9e1587fea7826bc5d5e640ee47a0bbf086b2a870179c5d58d60b0455b43d51d0 not found: ID does not exist" Sep 06 06:14:50.078481 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:50.078462 2578 scope.go:117] "RemoveContainer" containerID="510efe6d9e4f1d632b679fbab078c8b0d11ce369e6c0dce750bd30b40e517f36" Sep 06 06:14:50.078709 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:14:50.078692 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510efe6d9e4f1d632b679fbab078c8b0d11ce369e6c0dce750bd30b40e517f36\": container with ID starting with 510efe6d9e4f1d632b679fbab078c8b0d11ce369e6c0dce750bd30b40e517f36 not found: ID does not exist" containerID="510efe6d9e4f1d632b679fbab078c8b0d11ce369e6c0dce750bd30b40e517f36" Sep 06 06:14:50.078750 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:50.078716 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510efe6d9e4f1d632b679fbab078c8b0d11ce369e6c0dce750bd30b40e517f36"} err="failed to get container status \"510efe6d9e4f1d632b679fbab078c8b0d11ce369e6c0dce750bd30b40e517f36\": rpc error: code = NotFound desc = could not find container \"510efe6d9e4f1d632b679fbab078c8b0d11ce369e6c0dce750bd30b40e517f36\": container with ID starting with 510efe6d9e4f1d632b679fbab078c8b0d11ce369e6c0dce750bd30b40e517f36 not found: ID does not exist" Sep 06 06:14:50.084152 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:50.084128 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2"] Sep 06 06:14:50.088267 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:50.088246 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-c50b6f-openshift-default-849c48db4-6tds2"] Sep 06 06:14:51.144979 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:51.144942 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" path="/var/lib/kubelet/pods/5faad5e5-5edc-4d42-bc3d-012a8834d072/volumes" Sep 06 06:14:54.588510 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:14:54.588471 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:15:00.135556 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.135523 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29811255-5ml4h"] Sep 06 06:15:00.136071 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.135973 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerName="istio-proxy" Sep 06 06:15:00.136071 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.135986 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerName="istio-proxy" Sep 06 06:15:00.136071 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.136002 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerName="istio-proxy" Sep 06 06:15:00.136071 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.136011 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerName="istio-proxy" Sep 06 06:15:00.136217 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.136078 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerName="istio-proxy" Sep 06 06:15:00.136217 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.136095 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5faad5e5-5edc-4d42-bc3d-012a8834d072" containerName="istio-proxy" Sep 06 06:15:00.140666 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.140650 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ceca3b-29811255-cdhv2"] Sep 06 06:15:00.140774 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.140755 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811255-5ml4h" Sep 06 06:15:00.142806 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.142790 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:15:00.143924 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.143888 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ceca3b-29811255-cdhv2" Sep 06 06:15:00.147515 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.147489 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29811255-5ml4h"] Sep 06 06:15:00.150842 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.150344 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ceca3b-29811255-cdhv2"] Sep 06 06:15:00.156164 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.156144 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811255-5ml4h" Sep 06 06:15:00.164344 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.164317 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ceca3b-29811255-cdhv2" Sep 06 06:15:00.435161 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.435132 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29811255-5ml4h"] Sep 06 06:15:00.437499 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:15:00.437469 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61512e6_1f66_4c8b_8b17_d751660cda38.slice/crio-518f041477e121bf6d5910c0f558601035b18897a72d4b702839c3b45db401a3 WatchSource:0}: Error finding container 518f041477e121bf6d5910c0f558601035b18897a72d4b702839c3b45db401a3: Status 404 returned error can't find the container with id 518f041477e121bf6d5910c0f558601035b18897a72d4b702839c3b45db401a3 Sep 06 06:15:00.448763 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:00.448738 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ceca3b-29811255-cdhv2"] Sep 06 06:15:00.452084 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:15:00.452051 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1f84ba_0871_4115_b48d_a0ed4bcaf978.slice/crio-b1eda1baf0ea6030cbcece0c65f2e703e44cc01f5f378601e58cf9e3468f45a6 WatchSource:0}: Error finding container b1eda1baf0ea6030cbcece0c65f2e703e44cc01f5f378601e58cf9e3468f45a6: Status 404 returned error can't find the container with id b1eda1baf0ea6030cbcece0c65f2e703e44cc01f5f378601e58cf9e3468f45a6 Sep 06 06:15:01.122350 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:01.122306 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ceca3b-29811255-cdhv2" event={"ID":"8d1f84ba-0871-4115-b48d-a0ed4bcaf978","Type":"ContainerStarted","Data":"6e559a269130d273ca98571862468c908344da0056cc031284e1f62b093e660a"} Sep 06 06:15:01.122555 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:01.122367 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ceca3b-29811255-cdhv2" event={"ID":"8d1f84ba-0871-4115-b48d-a0ed4bcaf978","Type":"ContainerStarted","Data":"b1eda1baf0ea6030cbcece0c65f2e703e44cc01f5f378601e58cf9e3468f45a6"} Sep 06 06:15:01.124164 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:01.124138 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811255-5ml4h" event={"ID":"b61512e6-1f66-4c8b-8b17-d751660cda38","Type":"ContainerStarted","Data":"41fce9929cdf72706998668facfc2fd1664901d1a50cdc4a554e68afd2fb715e"} Sep 06 06:15:01.124284 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:01.124180 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811255-5ml4h" event={"ID":"b61512e6-1f66-4c8b-8b17-d751660cda38","Type":"ContainerStarted","Data":"518f041477e121bf6d5910c0f558601035b18897a72d4b702839c3b45db401a3"} Sep 06 06:15:01.137072 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:01.137024 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ceca3b-29811255-cdhv2" podStartSLOduration=1.137009831 podStartE2EDuration="1.137009831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:15:01.135096813 +0000 UTC m=+1266.533427300" watchObservedRunningTime="2026-09-06 06:15:01.137009831 +0000 UTC m=+1266.535340299" Sep 06 06:15:01.149955 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:01.149885 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811255-5ml4h" podStartSLOduration=1.149868482 podStartE2EDuration="1.149868482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:15:01.1473895 +0000 UTC m=+1266.545719967" watchObservedRunningTime="2026-09-06 06:15:01.149868482 +0000 UTC m=+1266.548198952" Sep 06 06:15:02.130750 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:02.130712 2578 generic.go:356] "Generic (PLEG): container finished" podID="8d1f84ba-0871-4115-b48d-a0ed4bcaf978" containerID="6e559a269130d273ca98571862468c908344da0056cc031284e1f62b093e660a" exitCode=0 Sep 06 06:15:02.130984 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:02.130809 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ceca3b-29811255-cdhv2" event={"ID":"8d1f84ba-0871-4115-b48d-a0ed4bcaf978","Type":"ContainerDied","Data":"6e559a269130d273ca98571862468c908344da0056cc031284e1f62b093e660a"} Sep 06 06:15:02.132437 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:02.132413 2578 generic.go:356] "Generic (PLEG): container finished" podID="b61512e6-1f66-4c8b-8b17-d751660cda38" containerID="41fce9929cdf72706998668facfc2fd1664901d1a50cdc4a554e68afd2fb715e" exitCode=0 Sep 06 06:15:02.132555 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:02.132438 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811255-5ml4h" event={"ID":"b61512e6-1f66-4c8b-8b17-d751660cda38","Type":"ContainerDied","Data":"41fce9929cdf72706998668facfc2fd1664901d1a50cdc4a554e68afd2fb715e"} Sep 06 06:15:03.313262 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:03.313233 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ceca3b-29811255-cdhv2" Sep 06 06:15:03.317272 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:03.317248 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811255-5ml4h" Sep 06 06:15:04.143966 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.143939 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ceca3b-29811255-cdhv2" Sep 06 06:15:04.144147 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.143973 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ceca3b-29811255-cdhv2" event={"ID":"8d1f84ba-0871-4115-b48d-a0ed4bcaf978","Type":"ContainerDied","Data":"b1eda1baf0ea6030cbcece0c65f2e703e44cc01f5f378601e58cf9e3468f45a6"} Sep 06 06:15:04.144147 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.144020 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1eda1baf0ea6030cbcece0c65f2e703e44cc01f5f378601e58cf9e3468f45a6" Sep 06 06:15:04.145322 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.145301 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811255-5ml4h" Sep 06 06:15:04.145427 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.145339 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811255-5ml4h" event={"ID":"b61512e6-1f66-4c8b-8b17-d751660cda38","Type":"ContainerDied","Data":"518f041477e121bf6d5910c0f558601035b18897a72d4b702839c3b45db401a3"} Sep 06 06:15:04.145427 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.145374 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="518f041477e121bf6d5910c0f558601035b18897a72d4b702839c3b45db401a3" Sep 06 06:15:04.265652 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.265620 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb"] Sep 06 06:15:04.266078 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.266064 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61512e6-1f66-4c8b-8b17-d751660cda38" containerName="cleanup" Sep 06 06:15:04.266128 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.266081 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61512e6-1f66-4c8b-8b17-d751660cda38" containerName="cleanup" Sep 06 06:15:04.266128 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.266094 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1f84ba-0871-4115-b48d-a0ed4bcaf978" containerName="cleanup" Sep 06 06:15:04.266128 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.266101 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1f84ba-0871-4115-b48d-a0ed4bcaf978" containerName="cleanup" Sep 06 06:15:04.266228 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.266179 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b61512e6-1f66-4c8b-8b17-d751660cda38" containerName="cleanup" Sep 06 06:15:04.266228 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.266196 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d1f84ba-0871-4115-b48d-a0ed4bcaf978" containerName="cleanup" Sep 06 06:15:04.269299 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.269281 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.271426 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.271406 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fc95779d-openshift-default-dockercfg-sll4g\"" Sep 06 06:15:04.282186 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.282157 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb"] Sep 06 06:15:04.356728 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.356655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-podinfo\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.356728 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.356705 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-workload-certs\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.359187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.356770 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-token\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.359187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.356859 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-credential-socket\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.359187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.356887 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-workload-socket\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.359187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.356947 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-envoy\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.359187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.357038 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzjgx\" (UniqueName: \"kubernetes.io/projected/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-kube-api-access-kzjgx\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.359187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.357380 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istiod-ca-cert\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.359187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.357543 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-data\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.458675 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.458576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-podinfo\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.458675 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.458620 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-workload-certs\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.458675 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.458673 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-token\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.459039 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.458710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-credential-socket\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.459039 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.458766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-workload-socket\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.459039 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.458824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-envoy\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.459039 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.458853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzjgx\" (UniqueName: \"kubernetes.io/projected/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-kube-api-access-kzjgx\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.459039 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.458896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istiod-ca-cert\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.459039 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.459003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-data\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.459327 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.459178 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-credential-socket\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.459327 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.459216 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-workload-socket\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.459500 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.459485 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-data\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.459634 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.459616 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istiod-ca-cert\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.459725 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.459710 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-workload-certs\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.461462 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.461440 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-podinfo\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.461674 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.461654 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-envoy\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.465609 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.465586 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-token\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.466002 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.465891 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzjgx\" (UniqueName: \"kubernetes.io/projected/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-kube-api-access-kzjgx\") pod \"e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.581385 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.581336 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:04.857329 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.857298 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb"] Sep 06 06:15:04.858720 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:15:04.858690 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a5cc1d_3dc7_4e4b_ae50_f05435be5f69.slice/crio-d75da2c3e04fb3d4455c9a5da85d1f40cecdd73812246d822b08e4ebf79dd7ff WatchSource:0}: Error finding container d75da2c3e04fb3d4455c9a5da85d1f40cecdd73812246d822b08e4ebf79dd7ff: Status 404 returned error can't find the container with id d75da2c3e04fb3d4455c9a5da85d1f40cecdd73812246d822b08e4ebf79dd7ff Sep 06 06:15:04.865954 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.862460 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:15:04.865954 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.862607 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:15:04.865954 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:04.862688 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:15:05.153547 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:05.153444 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" event={"ID":"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69","Type":"ContainerStarted","Data":"2ad8a914be0d04bde00a5f902e0a7d9f8ac4bfb87159481c66a44f7949cc1402"} Sep 06 06:15:05.153547 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:05.153515 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" event={"ID":"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69","Type":"ContainerStarted","Data":"d75da2c3e04fb3d4455c9a5da85d1f40cecdd73812246d822b08e4ebf79dd7ff"} Sep 06 06:15:05.173421 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:05.173365 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" podStartSLOduration=1.17334864 podStartE2EDuration="1.17334864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:15:05.171329673 +0000 UTC m=+1270.569660165" watchObservedRunningTime="2026-09-06 06:15:05.17334864 +0000 UTC m=+1270.571679107" Sep 06 06:15:05.582115 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:05.582069 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:05.587283 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:05.587259 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:06.158491 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:06.158463 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:06.159379 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:06.159359 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:09.086522 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:09.086482 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:15:16.714235 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:16.709109 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf"] Sep 06 06:15:16.714235 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:16.709473 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" podUID="1895bcbc-ea91-4458-b92c-de76f65b65ce" containerName="istio-proxy" containerID="cri-o://a951f7ef8c455860d93326ce41b5a878331a9e16b3dcc7a1eac492e73c67a3e7" gracePeriod=30 Sep 06 06:15:17.916185 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:17.916143 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49"] Sep 06 06:15:17.920300 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:17.920267 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49" Sep 06 06:15:17.926494 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:17.925400 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:15:17.926494 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:17.925732 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:15:17.927010 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:17.926990 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49"] Sep 06 06:15:17.997052 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:17.997016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4905728-0b17-4a90-83c9-2556a3613296-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49\" (UID: \"b4905728-0b17-4a90-83c9-2556a3613296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49" Sep 06 06:15:18.046319 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:18.046278 2578 patch_prober.go:28] interesting pod/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 06 06:15:18.046511 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:18.046363 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" podUID="1895bcbc-ea91-4458-b92c-de76f65b65ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 06 06:15:18.098270 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:18.098236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4905728-0b17-4a90-83c9-2556a3613296-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49\" (UID: \"b4905728-0b17-4a90-83c9-2556a3613296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49" Sep 06 06:15:18.098883 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:18.098861 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4905728-0b17-4a90-83c9-2556a3613296-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49\" (UID: \"b4905728-0b17-4a90-83c9-2556a3613296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49" Sep 06 06:15:18.243800 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:18.243692 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49" Sep 06 06:15:18.534047 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:18.534012 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49"] Sep 06 06:15:18.537802 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:15:18.537779 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4905728_0b17_4a90_83c9_2556a3613296.slice/crio-506da6722bc888e3196a4433d8c08c2381788490cf122942b954caf587819c4f WatchSource:0}: Error finding container 506da6722bc888e3196a4433d8c08c2381788490cf122942b954caf587819c4f: Status 404 returned error can't find the container with id 506da6722bc888e3196a4433d8c08c2381788490cf122942b954caf587819c4f Sep 06 06:15:19.240464 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:19.240367 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49" event={"ID":"b4905728-0b17-4a90-83c9-2556a3613296","Type":"ContainerStarted","Data":"20f982f7f42d7ce9fd699eacfccd58cd9549f60c3d7e0a1db2585d205213f346"} Sep 06 06:15:19.240464 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:19.240422 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49" event={"ID":"b4905728-0b17-4a90-83c9-2556a3613296","Type":"ContainerStarted","Data":"506da6722bc888e3196a4433d8c08c2381788490cf122942b954caf587819c4f"} Sep 06 06:15:19.254695 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:19.254625 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49" podStartSLOduration=2.254602872 podStartE2EDuration="2.254602872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:15:19.25297051 +0000 UTC m=+1284.651301004" watchObservedRunningTime="2026-09-06 06:15:19.254602872 +0000 UTC m=+1284.652933343" Sep 06 06:15:20.248014 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:20.247983 2578 generic.go:356] "Generic (PLEG): container finished" podID="b4905728-0b17-4a90-83c9-2556a3613296" containerID="20f982f7f42d7ce9fd699eacfccd58cd9549f60c3d7e0a1db2585d205213f346" exitCode=0 Sep 06 06:15:20.248484 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:20.248069 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49" event={"ID":"b4905728-0b17-4a90-83c9-2556a3613296","Type":"ContainerDied","Data":"20f982f7f42d7ce9fd699eacfccd58cd9549f60c3d7e0a1db2585d205213f346"} Sep 06 06:15:21.432573 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:21.432540 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49" Sep 06 06:15:21.536268 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:21.536235 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4905728-0b17-4a90-83c9-2556a3613296-maas-api-service-ca\") pod \"b4905728-0b17-4a90-83c9-2556a3613296\" (UID: \"b4905728-0b17-4a90-83c9-2556a3613296\") " Sep 06 06:15:21.536551 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:21.536528 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4905728-0b17-4a90-83c9-2556a3613296-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4905728-0b17-4a90-83c9-2556a3613296" (UID: "b4905728-0b17-4a90-83c9-2556a3613296"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:15:21.638107 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:21.638072 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4905728-0b17-4a90-83c9-2556a3613296-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:22.000779 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.000754 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:15:22.041865 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.041829 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-envoy\") pod \"1895bcbc-ea91-4458-b92c-de76f65b65ce\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " Sep 06 06:15:22.042084 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.041893 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-credential-socket\") pod \"1895bcbc-ea91-4458-b92c-de76f65b65ce\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " Sep 06 06:15:22.042084 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.041980 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-data\") pod \"1895bcbc-ea91-4458-b92c-de76f65b65ce\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " Sep 06 06:15:22.042084 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042060 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j7wh\" (UniqueName: \"kubernetes.io/projected/1895bcbc-ea91-4458-b92c-de76f65b65ce-kube-api-access-8j7wh\") pod \"1895bcbc-ea91-4458-b92c-de76f65b65ce\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " Sep 06 06:15:22.042271 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042127 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-workload-socket\") pod \"1895bcbc-ea91-4458-b92c-de76f65b65ce\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " Sep 06 06:15:22.042271 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042151 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-workload-certs\") pod \"1895bcbc-ea91-4458-b92c-de76f65b65ce\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " Sep 06 06:15:22.042271 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042201 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1895bcbc-ea91-4458-b92c-de76f65b65ce" (UID: "1895bcbc-ea91-4458-b92c-de76f65b65ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:22.042271 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042232 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "1895bcbc-ea91-4458-b92c-de76f65b65ce" (UID: "1895bcbc-ea91-4458-b92c-de76f65b65ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:22.042271 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042256 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-token\") pod \"1895bcbc-ea91-4458-b92c-de76f65b65ce\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " Sep 06 06:15:22.042514 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042289 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1895bcbc-ea91-4458-b92c-de76f65b65ce-istiod-ca-cert\") pod \"1895bcbc-ea91-4458-b92c-de76f65b65ce\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " Sep 06 06:15:22.042514 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042321 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-podinfo\") pod \"1895bcbc-ea91-4458-b92c-de76f65b65ce\" (UID: \"1895bcbc-ea91-4458-b92c-de76f65b65ce\") " Sep 06 06:15:22.042514 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042365 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1895bcbc-ea91-4458-b92c-de76f65b65ce" (UID: "1895bcbc-ea91-4458-b92c-de76f65b65ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:22.042514 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042440 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1895bcbc-ea91-4458-b92c-de76f65b65ce" (UID: "1895bcbc-ea91-4458-b92c-de76f65b65ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:22.042699 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042634 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1895bcbc-ea91-4458-b92c-de76f65b65ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1895bcbc-ea91-4458-b92c-de76f65b65ce" (UID: "1895bcbc-ea91-4458-b92c-de76f65b65ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:15:22.042781 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042763 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:22.042824 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042786 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:22.042824 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042801 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:22.042824 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042812 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:22.042992 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.042825 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1895bcbc-ea91-4458-b92c-de76f65b65ce-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:22.044985 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.044962 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1895bcbc-ea91-4458-b92c-de76f65b65ce" (UID: "1895bcbc-ea91-4458-b92c-de76f65b65ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:22.045099 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.045043 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1895bcbc-ea91-4458-b92c-de76f65b65ce-kube-api-access-8j7wh" (OuterVolumeSpecName: "kube-api-access-8j7wh") pod "1895bcbc-ea91-4458-b92c-de76f65b65ce" (UID: "1895bcbc-ea91-4458-b92c-de76f65b65ce"). InnerVolumeSpecName "kube-api-access-8j7wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:15:22.045201 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.045186 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1895bcbc-ea91-4458-b92c-de76f65b65ce" (UID: "1895bcbc-ea91-4458-b92c-de76f65b65ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:15:22.045298 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.045280 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "1895bcbc-ea91-4458-b92c-de76f65b65ce" (UID: "1895bcbc-ea91-4458-b92c-de76f65b65ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:15:22.143514 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.143439 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:22.143514 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.143467 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j7wh\" (UniqueName: \"kubernetes.io/projected/1895bcbc-ea91-4458-b92c-de76f65b65ce-kube-api-access-8j7wh\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:22.143514 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.143478 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:22.143514 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.143486 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1895bcbc-ea91-4458-b92c-de76f65b65ce-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:22.189722 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.189684 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:15:22.268225 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.268179 2578 generic.go:356] "Generic (PLEG): container finished" podID="1895bcbc-ea91-4458-b92c-de76f65b65ce" containerID="a951f7ef8c455860d93326ce41b5a878331a9e16b3dcc7a1eac492e73c67a3e7" exitCode=0 Sep 06 06:15:22.268439 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.268282 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" event={"ID":"1895bcbc-ea91-4458-b92c-de76f65b65ce","Type":"ContainerDied","Data":"a951f7ef8c455860d93326ce41b5a878331a9e16b3dcc7a1eac492e73c67a3e7"} Sep 06 06:15:22.268439 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.268340 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" event={"ID":"1895bcbc-ea91-4458-b92c-de76f65b65ce","Type":"ContainerDied","Data":"377124b201d12242c79d65903a5d03d4dde6d3f4d4ef22332db03e3ef669dc92"} Sep 06 06:15:22.268439 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.268373 2578 scope.go:117] "RemoveContainer" containerID="a951f7ef8c455860d93326ce41b5a878331a9e16b3dcc7a1eac492e73c67a3e7" Sep 06 06:15:22.268626 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.268528 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf" Sep 06 06:15:22.276810 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.276763 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49" event={"ID":"b4905728-0b17-4a90-83c9-2556a3613296","Type":"ContainerDied","Data":"506da6722bc888e3196a4433d8c08c2381788490cf122942b954caf587819c4f"} Sep 06 06:15:22.277058 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.277041 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="506da6722bc888e3196a4433d8c08c2381788490cf122942b954caf587819c4f" Sep 06 06:15:22.277169 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.277043 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49" Sep 06 06:15:22.287133 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.287105 2578 scope.go:117] "RemoveContainer" containerID="a951f7ef8c455860d93326ce41b5a878331a9e16b3dcc7a1eac492e73c67a3e7" Sep 06 06:15:22.287484 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:15:22.287463 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a951f7ef8c455860d93326ce41b5a878331a9e16b3dcc7a1eac492e73c67a3e7\": container with ID starting with a951f7ef8c455860d93326ce41b5a878331a9e16b3dcc7a1eac492e73c67a3e7 not found: ID does not exist" containerID="a951f7ef8c455860d93326ce41b5a878331a9e16b3dcc7a1eac492e73c67a3e7" Sep 06 06:15:22.287565 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.287497 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a951f7ef8c455860d93326ce41b5a878331a9e16b3dcc7a1eac492e73c67a3e7"} err="failed to get container status \"a951f7ef8c455860d93326ce41b5a878331a9e16b3dcc7a1eac492e73c67a3e7\": rpc error: code = NotFound desc = could not find container \"a951f7ef8c455860d93326ce41b5a878331a9e16b3dcc7a1eac492e73c67a3e7\": container with ID starting with a951f7ef8c455860d93326ce41b5a878331a9e16b3dcc7a1eac492e73c67a3e7 not found: ID does not exist" Sep 06 06:15:22.295672 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.295638 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf"] Sep 06 06:15:22.301642 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:22.301615 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-837ae9-openshift-default-8cf499cc8-9n6sf"] Sep 06 06:15:23.145346 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:23.145314 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1895bcbc-ea91-4458-b92c-de76f65b65ce" path="/var/lib/kubelet/pods/1895bcbc-ea91-4458-b92c-de76f65b65ce/volumes" Sep 06 06:15:27.310597 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:27.310562 2578 generic.go:356] "Generic (PLEG): container finished" podID="e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" containerID="2ad8a914be0d04bde00a5f902e0a7d9f8ac4bfb87159481c66a44f7949cc1402" exitCode=0 Sep 06 06:15:27.311039 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:27.310632 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" event={"ID":"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69","Type":"ContainerDied","Data":"2ad8a914be0d04bde00a5f902e0a7d9f8ac4bfb87159481c66a44f7949cc1402"} Sep 06 06:15:27.311137 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:27.311121 2578 scope.go:117] "RemoveContainer" containerID="2ad8a914be0d04bde00a5f902e0a7d9f8ac4bfb87159481c66a44f7949cc1402" Sep 06 06:15:27.312316 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:27.312280 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:15:27.312390 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:27.312373 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:15:27.312428 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:27.312409 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:15:28.319714 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:28.319681 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" event={"ID":"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69","Type":"ContainerStarted","Data":"59dda6f7ca462cf6919dcf864cf9aa7b7045375ea6504eb7c9aabf052f0aa854"} Sep 06 06:15:28.369934 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:28.366425 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ceca3b-29811255-cdhv2"] Sep 06 06:15:28.379957 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:28.379039 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ceca3b-29811255-cdhv2"] Sep 06 06:15:28.535522 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:28.535459 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49"] Sep 06 06:15:28.537575 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:28.537548 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ceca3b-rxx49"] Sep 06 06:15:28.581741 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:28.581658 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:28.581953 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:28.581879 2578 patch_prober.go:28] interesting pod/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 06 06:15:28.582014 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:28.581980 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" podUID="e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 06 06:15:29.146866 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:29.146825 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d1f84ba-0871-4115-b48d-a0ed4bcaf978" path="/var/lib/kubelet/pods/8d1f84ba-0871-4115-b48d-a0ed4bcaf978/volumes" Sep 06 06:15:29.147194 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:29.147179 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4905728-0b17-4a90-83c9-2556a3613296" path="/var/lib/kubelet/pods/b4905728-0b17-4a90-83c9-2556a3613296/volumes" Sep 06 06:15:29.581894 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:29.581854 2578 patch_prober.go:28] interesting pod/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 06 06:15:29.582333 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:29.581948 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" podUID="e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 06 06:15:30.582104 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:30.582061 2578 patch_prober.go:28] interesting pod/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 06 06:15:30.582628 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:30.582136 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" podUID="e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 06 06:15:31.194303 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:31.194265 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:15:31.582626 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:31.582586 2578 patch_prober.go:28] interesting pod/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 06 06:15:31.583089 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:31.582650 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" podUID="e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 06 06:15:32.582418 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:32.582375 2578 patch_prober.go:28] interesting pod/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 06 06:15:32.582615 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:32.582434 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" podUID="e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 06 06:15:33.719795 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:33.719765 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:33.720305 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:33.720244 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:33.721733 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:33.721712 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:15:35.003963 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.003933 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm"] Sep 06 06:15:35.004426 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.004185 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" podUID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerName="istio-proxy" containerID="cri-o://3805697d131e86a9ea7fd3332aa3b1585c9f5929b5be3a1d0da92a9c4c035263" gracePeriod=30 Sep 06 06:15:35.331155 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.331122 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk"] Sep 06 06:15:35.331511 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.331500 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4905728-0b17-4a90-83c9-2556a3613296" containerName="revoke-keys" Sep 06 06:15:35.331558 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.331514 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4905728-0b17-4a90-83c9-2556a3613296" containerName="revoke-keys" Sep 06 06:15:35.331558 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.331533 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1895bcbc-ea91-4458-b92c-de76f65b65ce" containerName="istio-proxy" Sep 06 06:15:35.331558 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.331538 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1895bcbc-ea91-4458-b92c-de76f65b65ce" containerName="istio-proxy" Sep 06 06:15:35.331659 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.331599 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1895bcbc-ea91-4458-b92c-de76f65b65ce" containerName="istio-proxy" Sep 06 06:15:35.331659 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.331613 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4905728-0b17-4a90-83c9-2556a3613296" containerName="revoke-keys" Sep 06 06:15:35.336004 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.335985 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk" Sep 06 06:15:35.338160 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.338139 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:15:35.338270 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.338197 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:15:35.351426 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.351374 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk"] Sep 06 06:15:35.492411 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.492366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42da1201-996b-471a-8270-34831ce67996-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk\" (UID: \"42da1201-996b-471a-8270-34831ce67996\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk" Sep 06 06:15:35.594116 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.594016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42da1201-996b-471a-8270-34831ce67996-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk\" (UID: \"42da1201-996b-471a-8270-34831ce67996\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk" Sep 06 06:15:35.594666 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.594644 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42da1201-996b-471a-8270-34831ce67996-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk\" (UID: \"42da1201-996b-471a-8270-34831ce67996\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk" Sep 06 06:15:35.653045 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.653006 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk" Sep 06 06:15:35.923188 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:35.923160 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk"] Sep 06 06:15:35.925135 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:15:35.925108 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42da1201_996b_471a_8270_34831ce67996.slice/crio-696e0a40f0f20a8f10c51017ecde9c3aad817fe11cda0b2ddc054dde57bd11ec WatchSource:0}: Error finding container 696e0a40f0f20a8f10c51017ecde9c3aad817fe11cda0b2ddc054dde57bd11ec: Status 404 returned error can't find the container with id 696e0a40f0f20a8f10c51017ecde9c3aad817fe11cda0b2ddc054dde57bd11ec Sep 06 06:15:36.132239 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:36.132200 2578 patch_prober.go:28] interesting pod/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 06 06:15:36.132612 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:36.132272 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" podUID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 06 06:15:36.366101 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:36.366063 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk" event={"ID":"42da1201-996b-471a-8270-34831ce67996","Type":"ContainerStarted","Data":"b5ea4ec3413bd3acf49ef3f3c98365ff022c5f3224836a26f01250a19117b01b"} Sep 06 06:15:36.366286 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:36.366119 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk" event={"ID":"42da1201-996b-471a-8270-34831ce67996","Type":"ContainerStarted","Data":"696e0a40f0f20a8f10c51017ecde9c3aad817fe11cda0b2ddc054dde57bd11ec"} Sep 06 06:15:36.380508 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:36.380456 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk" podStartSLOduration=1.380440203 podStartE2EDuration="1.380440203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:15:36.377935875 +0000 UTC m=+1301.776266599" watchObservedRunningTime="2026-09-06 06:15:36.380440203 +0000 UTC m=+1301.778770671" Sep 06 06:15:37.373050 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:37.373013 2578 generic.go:356] "Generic (PLEG): container finished" podID="42da1201-996b-471a-8270-34831ce67996" containerID="b5ea4ec3413bd3acf49ef3f3c98365ff022c5f3224836a26f01250a19117b01b" exitCode=0 Sep 06 06:15:37.373441 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:37.373095 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk" event={"ID":"42da1201-996b-471a-8270-34831ce67996","Type":"ContainerDied","Data":"b5ea4ec3413bd3acf49ef3f3c98365ff022c5f3224836a26f01250a19117b01b"} Sep 06 06:15:38.552842 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:38.552815 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk" Sep 06 06:15:38.727105 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:38.727018 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42da1201-996b-471a-8270-34831ce67996-maas-api-service-ca\") pod \"42da1201-996b-471a-8270-34831ce67996\" (UID: \"42da1201-996b-471a-8270-34831ce67996\") " Sep 06 06:15:38.727405 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:38.727378 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42da1201-996b-471a-8270-34831ce67996-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42da1201-996b-471a-8270-34831ce67996" (UID: "42da1201-996b-471a-8270-34831ce67996"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:15:38.828582 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:38.828550 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42da1201-996b-471a-8270-34831ce67996-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:39.160200 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:39.160168 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j"] Sep 06 06:15:39.160592 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:39.160580 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42da1201-996b-471a-8270-34831ce67996" containerName="revoke-keys" Sep 06 06:15:39.160639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:39.160594 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="42da1201-996b-471a-8270-34831ce67996" containerName="revoke-keys" Sep 06 06:15:39.160674 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:39.160665 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="42da1201-996b-471a-8270-34831ce67996" containerName="revoke-keys" Sep 06 06:15:39.164032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:39.164006 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j" Sep 06 06:15:39.170285 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:39.170023 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j"] Sep 06 06:15:39.333779 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:39.333748 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebab2f82-2062-46dc-9818-9292a4617b12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j\" (UID: \"ebab2f82-2062-46dc-9818-9292a4617b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j" Sep 06 06:15:39.392987 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:39.392948 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk" event={"ID":"42da1201-996b-471a-8270-34831ce67996","Type":"ContainerDied","Data":"696e0a40f0f20a8f10c51017ecde9c3aad817fe11cda0b2ddc054dde57bd11ec"} Sep 06 06:15:39.392987 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:39.392980 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk" Sep 06 06:15:39.393245 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:39.393037 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="696e0a40f0f20a8f10c51017ecde9c3aad817fe11cda0b2ddc054dde57bd11ec" Sep 06 06:15:39.435409 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:39.435318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebab2f82-2062-46dc-9818-9292a4617b12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j\" (UID: \"ebab2f82-2062-46dc-9818-9292a4617b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j" Sep 06 06:15:39.436008 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:39.435987 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebab2f82-2062-46dc-9818-9292a4617b12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j\" (UID: \"ebab2f82-2062-46dc-9818-9292a4617b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j" Sep 06 06:15:39.477599 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:39.477562 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j" Sep 06 06:15:39.752256 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:39.752220 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j"] Sep 06 06:15:39.753025 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:15:39.753000 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebab2f82_2062_46dc_9818_9292a4617b12.slice/crio-ce7263a4778a418ea30054137faf7c41dedb3657038bf6edb1497a2f5bbee2c8 WatchSource:0}: Error finding container ce7263a4778a418ea30054137faf7c41dedb3657038bf6edb1497a2f5bbee2c8: Status 404 returned error can't find the container with id ce7263a4778a418ea30054137faf7c41dedb3657038bf6edb1497a2f5bbee2c8 Sep 06 06:15:40.320516 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.320485 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:15:40.401893 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.401857 2578 generic.go:356] "Generic (PLEG): container finished" podID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerID="3805697d131e86a9ea7fd3332aa3b1585c9f5929b5be3a1d0da92a9c4c035263" exitCode=0 Sep 06 06:15:40.402146 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.401949 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" Sep 06 06:15:40.402146 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.401940 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" event={"ID":"cd5046cd-6f68-45bf-8e6b-59444f9af36d","Type":"ContainerDied","Data":"3805697d131e86a9ea7fd3332aa3b1585c9f5929b5be3a1d0da92a9c4c035263"} Sep 06 06:15:40.402146 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.402082 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm" event={"ID":"cd5046cd-6f68-45bf-8e6b-59444f9af36d","Type":"ContainerDied","Data":"f07e5d8ff5a54cb8bb5001e6de6bc3d7b3bf3c0ae8bdff697ef518e177c5395d"} Sep 06 06:15:40.402146 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.402109 2578 scope.go:117] "RemoveContainer" containerID="3805697d131e86a9ea7fd3332aa3b1585c9f5929b5be3a1d0da92a9c4c035263" Sep 06 06:15:40.404005 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.403961 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j" event={"ID":"ebab2f82-2062-46dc-9818-9292a4617b12","Type":"ContainerStarted","Data":"490622ecc70a87ad5a124b2084bbecb47f4fc7b30d661c8a859efc0acc91c94a"} Sep 06 06:15:40.404130 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.404055 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j" event={"ID":"ebab2f82-2062-46dc-9818-9292a4617b12","Type":"ContainerStarted","Data":"ce7263a4778a418ea30054137faf7c41dedb3657038bf6edb1497a2f5bbee2c8"} Sep 06 06:15:40.411982 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.411957 2578 scope.go:117] "RemoveContainer" containerID="f86d4da5d4968e152942f0e3f0b37e044c235b256d31bb13d8a3d5312279afb9" Sep 06 06:15:40.423362 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.423306 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j" podStartSLOduration=1.422960319 podStartE2EDuration="1.422960319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:15:40.421290505 +0000 UTC m=+1305.819620974" watchObservedRunningTime="2026-09-06 06:15:40.422960319 +0000 UTC m=+1305.821290787" Sep 06 06:15:40.425984 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.425944 2578 scope.go:117] "RemoveContainer" containerID="3805697d131e86a9ea7fd3332aa3b1585c9f5929b5be3a1d0da92a9c4c035263" Sep 06 06:15:40.426255 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:15:40.426233 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3805697d131e86a9ea7fd3332aa3b1585c9f5929b5be3a1d0da92a9c4c035263\": container with ID starting with 3805697d131e86a9ea7fd3332aa3b1585c9f5929b5be3a1d0da92a9c4c035263 not found: ID does not exist" containerID="3805697d131e86a9ea7fd3332aa3b1585c9f5929b5be3a1d0da92a9c4c035263" Sep 06 06:15:40.426341 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.426268 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3805697d131e86a9ea7fd3332aa3b1585c9f5929b5be3a1d0da92a9c4c035263"} err="failed to get container status \"3805697d131e86a9ea7fd3332aa3b1585c9f5929b5be3a1d0da92a9c4c035263\": rpc error: code = NotFound desc = could not find container \"3805697d131e86a9ea7fd3332aa3b1585c9f5929b5be3a1d0da92a9c4c035263\": container with ID starting with 3805697d131e86a9ea7fd3332aa3b1585c9f5929b5be3a1d0da92a9c4c035263 not found: ID does not exist" Sep 06 06:15:40.426341 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.426291 2578 scope.go:117] "RemoveContainer" containerID="f86d4da5d4968e152942f0e3f0b37e044c235b256d31bb13d8a3d5312279afb9" Sep 06 06:15:40.426571 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:15:40.426552 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86d4da5d4968e152942f0e3f0b37e044c235b256d31bb13d8a3d5312279afb9\": container with ID starting with f86d4da5d4968e152942f0e3f0b37e044c235b256d31bb13d8a3d5312279afb9 not found: ID does not exist" containerID="f86d4da5d4968e152942f0e3f0b37e044c235b256d31bb13d8a3d5312279afb9" Sep 06 06:15:40.426626 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.426577 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86d4da5d4968e152942f0e3f0b37e044c235b256d31bb13d8a3d5312279afb9"} err="failed to get container status \"f86d4da5d4968e152942f0e3f0b37e044c235b256d31bb13d8a3d5312279afb9\": rpc error: code = NotFound desc = could not find container \"f86d4da5d4968e152942f0e3f0b37e044c235b256d31bb13d8a3d5312279afb9\": container with ID starting with f86d4da5d4968e152942f0e3f0b37e044c235b256d31bb13d8a3d5312279afb9 not found: ID does not exist" Sep 06 06:15:40.448116 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.446966 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-workload-certs\") pod \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " Sep 06 06:15:40.448116 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.447029 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-podinfo\") pod \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " Sep 06 06:15:40.448116 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.447062 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-data\") pod \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " Sep 06 06:15:40.448116 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.447090 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-token\") pod \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " Sep 06 06:15:40.448116 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.447118 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s994\" (UniqueName: \"kubernetes.io/projected/cd5046cd-6f68-45bf-8e6b-59444f9af36d-kube-api-access-2s994\") pod \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " Sep 06 06:15:40.448116 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.447176 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-credential-socket\") pod \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " Sep 06 06:15:40.448116 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.447234 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istiod-ca-cert\") pod \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " Sep 06 06:15:40.448116 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.447338 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-workload-socket\") pod \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " Sep 06 06:15:40.448116 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.447371 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-envoy\") pod \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\" (UID: \"cd5046cd-6f68-45bf-8e6b-59444f9af36d\") " Sep 06 06:15:40.451658 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.451435 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd5046cd-6f68-45bf-8e6b-59444f9af36d" (UID: "cd5046cd-6f68-45bf-8e6b-59444f9af36d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:40.452176 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.451827 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd5046cd-6f68-45bf-8e6b-59444f9af36d" (UID: "cd5046cd-6f68-45bf-8e6b-59444f9af36d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:40.455256 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.455074 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd5046cd-6f68-45bf-8e6b-59444f9af36d" (UID: "cd5046cd-6f68-45bf-8e6b-59444f9af36d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:15:40.455585 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.455406 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd5046cd-6f68-45bf-8e6b-59444f9af36d" (UID: "cd5046cd-6f68-45bf-8e6b-59444f9af36d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:40.456047 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.455809 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd5046cd-6f68-45bf-8e6b-59444f9af36d" (UID: "cd5046cd-6f68-45bf-8e6b-59444f9af36d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:15:40.456347 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.456179 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd5046cd-6f68-45bf-8e6b-59444f9af36d" (UID: "cd5046cd-6f68-45bf-8e6b-59444f9af36d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:40.456722 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.456470 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd5046cd-6f68-45bf-8e6b-59444f9af36d" (UID: "cd5046cd-6f68-45bf-8e6b-59444f9af36d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:40.458929 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.458870 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5046cd-6f68-45bf-8e6b-59444f9af36d-kube-api-access-2s994" (OuterVolumeSpecName: "kube-api-access-2s994") pod "cd5046cd-6f68-45bf-8e6b-59444f9af36d" (UID: "cd5046cd-6f68-45bf-8e6b-59444f9af36d"). InnerVolumeSpecName "kube-api-access-2s994". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:15:40.461561 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.461525 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd5046cd-6f68-45bf-8e6b-59444f9af36d" (UID: "cd5046cd-6f68-45bf-8e6b-59444f9af36d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:15:40.549147 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.549057 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:40.549147 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.549095 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:40.549147 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.549113 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:40.549147 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.549127 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:40.549147 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.549141 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:40.549147 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.549153 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:40.549465 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.549166 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s994\" (UniqueName: \"kubernetes.io/projected/cd5046cd-6f68-45bf-8e6b-59444f9af36d-kube-api-access-2s994\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:40.549465 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.549184 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5046cd-6f68-45bf-8e6b-59444f9af36d-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:40.549465 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.549197 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd5046cd-6f68-45bf-8e6b-59444f9af36d-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:40.731159 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.731112 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm"] Sep 06 06:15:40.733668 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:40.733646 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ceca3b-openshift-default-68fd786bc6-j88wm"] Sep 06 06:15:41.147019 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:41.146983 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" path="/var/lib/kubelet/pods/cd5046cd-6f68-45bf-8e6b-59444f9af36d/volumes" Sep 06 06:15:41.415938 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:41.415814 2578 generic.go:356] "Generic (PLEG): container finished" podID="ebab2f82-2062-46dc-9818-9292a4617b12" containerID="490622ecc70a87ad5a124b2084bbecb47f4fc7b30d661c8a859efc0acc91c94a" exitCode=0 Sep 06 06:15:41.415938 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:41.415865 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j" event={"ID":"ebab2f82-2062-46dc-9818-9292a4617b12","Type":"ContainerDied","Data":"490622ecc70a87ad5a124b2084bbecb47f4fc7b30d661c8a859efc0acc91c94a"} Sep 06 06:15:42.613280 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:42.613248 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j" Sep 06 06:15:42.774077 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:42.773326 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebab2f82-2062-46dc-9818-9292a4617b12-maas-api-service-ca\") pod \"ebab2f82-2062-46dc-9818-9292a4617b12\" (UID: \"ebab2f82-2062-46dc-9818-9292a4617b12\") " Sep 06 06:15:42.774077 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:42.774025 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebab2f82-2062-46dc-9818-9292a4617b12-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebab2f82-2062-46dc-9818-9292a4617b12" (UID: "ebab2f82-2062-46dc-9818-9292a4617b12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:15:42.874409 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:42.874377 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebab2f82-2062-46dc-9818-9292a4617b12-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:43.429123 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:43.429027 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j" Sep 06 06:15:43.429123 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:43.429033 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j" event={"ID":"ebab2f82-2062-46dc-9818-9292a4617b12","Type":"ContainerDied","Data":"ce7263a4778a418ea30054137faf7c41dedb3657038bf6edb1497a2f5bbee2c8"} Sep 06 06:15:43.429123 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:43.429089 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce7263a4778a418ea30054137faf7c41dedb3657038bf6edb1497a2f5bbee2c8" Sep 06 06:15:46.267164 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:46.267129 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk"] Sep 06 06:15:46.269993 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:46.269958 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0bce0b-26jrk"] Sep 06 06:15:47.145019 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:47.144984 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42da1201-996b-471a-8270-34831ce67996" path="/var/lib/kubelet/pods/42da1201-996b-471a-8270-34831ce67996/volumes" Sep 06 06:15:49.989778 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:49.989746 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:15:50.127937 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:50.127870 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j"] Sep 06 06:15:50.129766 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:50.129738 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc95779d-cnr4j"] Sep 06 06:15:51.145394 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:51.145360 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebab2f82-2062-46dc-9818-9292a4617b12" path="/var/lib/kubelet/pods/ebab2f82-2062-46dc-9818-9292a4617b12/volumes" Sep 06 06:15:52.553601 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:52.553569 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp"] Sep 06 06:15:52.554091 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:52.553833 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" podUID="452e293c-1514-4cca-b4c8-d0d4e10a0290" containerName="istio-proxy" containerID="cri-o://861a32240a772cef8c790e3ba0ea9d434907ff45c38d3ea1a55f189ee3551692" gracePeriod=30 Sep 06 06:15:56.235669 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:56.235625 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb"] Sep 06 06:15:56.236093 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:56.235969 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" podUID="e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" containerName="istio-proxy" containerID="cri-o://59dda6f7ca462cf6919dcf864cf9aa7b7045375ea6504eb7c9aabf052f0aa854" gracePeriod=30 Sep 06 06:15:57.812243 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.812219 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:15:57.925338 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.925242 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-credential-socket\") pod \"452e293c-1514-4cca-b4c8-d0d4e10a0290\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " Sep 06 06:15:57.925338 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.925299 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452e293c-1514-4cca-b4c8-d0d4e10a0290-istiod-ca-cert\") pod \"452e293c-1514-4cca-b4c8-d0d4e10a0290\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " Sep 06 06:15:57.925338 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.925318 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-246zx\" (UniqueName: \"kubernetes.io/projected/452e293c-1514-4cca-b4c8-d0d4e10a0290-kube-api-access-246zx\") pod \"452e293c-1514-4cca-b4c8-d0d4e10a0290\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " Sep 06 06:15:57.925621 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.925372 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-workload-socket\") pod \"452e293c-1514-4cca-b4c8-d0d4e10a0290\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " Sep 06 06:15:57.925621 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.925409 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-data\") pod \"452e293c-1514-4cca-b4c8-d0d4e10a0290\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " Sep 06 06:15:57.925621 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.925429 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-token\") pod \"452e293c-1514-4cca-b4c8-d0d4e10a0290\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " Sep 06 06:15:57.925621 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.925534 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-envoy\") pod \"452e293c-1514-4cca-b4c8-d0d4e10a0290\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " Sep 06 06:15:57.925621 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.925597 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-workload-certs\") pod \"452e293c-1514-4cca-b4c8-d0d4e10a0290\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " Sep 06 06:15:57.925621 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.925574 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "452e293c-1514-4cca-b4c8-d0d4e10a0290" (UID: "452e293c-1514-4cca-b4c8-d0d4e10a0290"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:57.925963 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.925638 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-podinfo\") pod \"452e293c-1514-4cca-b4c8-d0d4e10a0290\" (UID: \"452e293c-1514-4cca-b4c8-d0d4e10a0290\") " Sep 06 06:15:57.925963 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.925734 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/452e293c-1514-4cca-b4c8-d0d4e10a0290-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "452e293c-1514-4cca-b4c8-d0d4e10a0290" (UID: "452e293c-1514-4cca-b4c8-d0d4e10a0290"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:15:57.925963 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.925778 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "452e293c-1514-4cca-b4c8-d0d4e10a0290" (UID: "452e293c-1514-4cca-b4c8-d0d4e10a0290"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:57.925963 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.925947 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "452e293c-1514-4cca-b4c8-d0d4e10a0290" (UID: "452e293c-1514-4cca-b4c8-d0d4e10a0290"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:57.926149 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.926074 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:57.926149 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.926091 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:57.926149 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.926104 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:57.926149 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.926116 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452e293c-1514-4cca-b4c8-d0d4e10a0290-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:57.926276 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.926214 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-data" (OuterVolumeSpecName: "istio-data") pod "452e293c-1514-4cca-b4c8-d0d4e10a0290" (UID: "452e293c-1514-4cca-b4c8-d0d4e10a0290"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:57.928020 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.927990 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "452e293c-1514-4cca-b4c8-d0d4e10a0290" (UID: "452e293c-1514-4cca-b4c8-d0d4e10a0290"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:15:57.928020 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.928008 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "452e293c-1514-4cca-b4c8-d0d4e10a0290" (UID: "452e293c-1514-4cca-b4c8-d0d4e10a0290"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:15:57.928238 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.928223 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452e293c-1514-4cca-b4c8-d0d4e10a0290-kube-api-access-246zx" (OuterVolumeSpecName: "kube-api-access-246zx") pod "452e293c-1514-4cca-b4c8-d0d4e10a0290" (UID: "452e293c-1514-4cca-b4c8-d0d4e10a0290"). InnerVolumeSpecName "kube-api-access-246zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:15:57.928371 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:57.928349 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-token" (OuterVolumeSpecName: "istio-token") pod "452e293c-1514-4cca-b4c8-d0d4e10a0290" (UID: "452e293c-1514-4cca-b4c8-d0d4e10a0290"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:15:58.027012 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.026980 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-246zx\" (UniqueName: \"kubernetes.io/projected/452e293c-1514-4cca-b4c8-d0d4e10a0290-kube-api-access-246zx\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:58.027012 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.027007 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:58.027012 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.027019 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:58.027207 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.027026 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:58.027207 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.027034 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452e293c-1514-4cca-b4c8-d0d4e10a0290-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:15:58.524878 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.524842 2578 generic.go:356] "Generic (PLEG): container finished" podID="452e293c-1514-4cca-b4c8-d0d4e10a0290" containerID="861a32240a772cef8c790e3ba0ea9d434907ff45c38d3ea1a55f189ee3551692" exitCode=0 Sep 06 06:15:58.525081 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.524892 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" Sep 06 06:15:58.525081 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.524932 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" event={"ID":"452e293c-1514-4cca-b4c8-d0d4e10a0290","Type":"ContainerDied","Data":"861a32240a772cef8c790e3ba0ea9d434907ff45c38d3ea1a55f189ee3551692"} Sep 06 06:15:58.525081 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.524990 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp" event={"ID":"452e293c-1514-4cca-b4c8-d0d4e10a0290","Type":"ContainerDied","Data":"3e3f541a3ce17d92d2db3680b5188bbeadad3c24d22d80f621f3b3499ea241db"} Sep 06 06:15:58.525081 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.525012 2578 scope.go:117] "RemoveContainer" containerID="861a32240a772cef8c790e3ba0ea9d434907ff45c38d3ea1a55f189ee3551692" Sep 06 06:15:58.535069 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.535053 2578 scope.go:117] "RemoveContainer" containerID="a86d10d4e6d40648ffa93bcff3fe0942f0d47d11eb84d948a5944985d5d4dbbd" Sep 06 06:15:58.546242 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.545273 2578 scope.go:117] "RemoveContainer" containerID="861a32240a772cef8c790e3ba0ea9d434907ff45c38d3ea1a55f189ee3551692" Sep 06 06:15:58.546859 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:15:58.546833 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"861a32240a772cef8c790e3ba0ea9d434907ff45c38d3ea1a55f189ee3551692\": container with ID starting with 861a32240a772cef8c790e3ba0ea9d434907ff45c38d3ea1a55f189ee3551692 not found: ID does not exist" containerID="861a32240a772cef8c790e3ba0ea9d434907ff45c38d3ea1a55f189ee3551692" Sep 06 06:15:58.547012 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.546872 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"861a32240a772cef8c790e3ba0ea9d434907ff45c38d3ea1a55f189ee3551692"} err="failed to get container status \"861a32240a772cef8c790e3ba0ea9d434907ff45c38d3ea1a55f189ee3551692\": rpc error: code = NotFound desc = could not find container \"861a32240a772cef8c790e3ba0ea9d434907ff45c38d3ea1a55f189ee3551692\": container with ID starting with 861a32240a772cef8c790e3ba0ea9d434907ff45c38d3ea1a55f189ee3551692 not found: ID does not exist" Sep 06 06:15:58.547012 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.546898 2578 scope.go:117] "RemoveContainer" containerID="a86d10d4e6d40648ffa93bcff3fe0942f0d47d11eb84d948a5944985d5d4dbbd" Sep 06 06:15:58.547304 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:15:58.547281 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86d10d4e6d40648ffa93bcff3fe0942f0d47d11eb84d948a5944985d5d4dbbd\": container with ID starting with a86d10d4e6d40648ffa93bcff3fe0942f0d47d11eb84d948a5944985d5d4dbbd not found: ID does not exist" containerID="a86d10d4e6d40648ffa93bcff3fe0942f0d47d11eb84d948a5944985d5d4dbbd" Sep 06 06:15:58.547349 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.547316 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86d10d4e6d40648ffa93bcff3fe0942f0d47d11eb84d948a5944985d5d4dbbd"} err="failed to get container status \"a86d10d4e6d40648ffa93bcff3fe0942f0d47d11eb84d948a5944985d5d4dbbd\": rpc error: code = NotFound desc = could not find container \"a86d10d4e6d40648ffa93bcff3fe0942f0d47d11eb84d948a5944985d5d4dbbd\": container with ID starting with a86d10d4e6d40648ffa93bcff3fe0942f0d47d11eb84d948a5944985d5d4dbbd not found: ID does not exist" Sep 06 06:15:58.548479 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.548443 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp"] Sep 06 06:15:58.550351 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:58.550330 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0bce0b-openshift-default-68c6676c44-m2kwp"] Sep 06 06:15:59.082453 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:59.082407 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:15:59.145132 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:15:59.145102 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="452e293c-1514-4cca-b4c8-d0d4e10a0290" path="/var/lib/kubelet/pods/452e293c-1514-4cca-b4c8-d0d4e10a0290/volumes" Sep 06 06:16:01.500104 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.500042 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:16:01.544243 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.544210 2578 generic.go:356] "Generic (PLEG): container finished" podID="e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" containerID="59dda6f7ca462cf6919dcf864cf9aa7b7045375ea6504eb7c9aabf052f0aa854" exitCode=0 Sep 06 06:16:01.544420 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.544273 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" Sep 06 06:16:01.544420 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.544301 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" event={"ID":"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69","Type":"ContainerDied","Data":"59dda6f7ca462cf6919dcf864cf9aa7b7045375ea6504eb7c9aabf052f0aa854"} Sep 06 06:16:01.544420 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.544390 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb" event={"ID":"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69","Type":"ContainerDied","Data":"d75da2c3e04fb3d4455c9a5da85d1f40cecdd73812246d822b08e4ebf79dd7ff"} Sep 06 06:16:01.544420 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.544415 2578 scope.go:117] "RemoveContainer" containerID="59dda6f7ca462cf6919dcf864cf9aa7b7045375ea6504eb7c9aabf052f0aa854" Sep 06 06:16:01.553575 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.553554 2578 scope.go:117] "RemoveContainer" containerID="2ad8a914be0d04bde00a5f902e0a7d9f8ac4bfb87159481c66a44f7949cc1402" Sep 06 06:16:01.556832 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.556811 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istiod-ca-cert\") pod \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " Sep 06 06:16:01.556935 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.556853 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-data\") pod \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " Sep 06 06:16:01.556935 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.556887 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-workload-certs\") pod \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " Sep 06 06:16:01.557050 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.556944 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-credential-socket\") pod \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " Sep 06 06:16:01.557050 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.556971 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-podinfo\") pod \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " Sep 06 06:16:01.557050 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.557003 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-envoy\") pod \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " Sep 06 06:16:01.557208 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.557067 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-token\") pod \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " Sep 06 06:16:01.557208 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.557096 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzjgx\" (UniqueName: \"kubernetes.io/projected/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-kube-api-access-kzjgx\") pod \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " Sep 06 06:16:01.557208 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.557131 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-data" (OuterVolumeSpecName: "istio-data") pod "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" (UID: "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:16:01.557208 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.557139 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" (UID: "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:16:01.557208 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.557155 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-workload-socket\") pod \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\" (UID: \"e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69\") " Sep 06 06:16:01.557443 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.557360 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" (UID: "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:16:01.557443 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.557370 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" (UID: "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:16:01.557573 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.557545 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:16:01.557638 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.557579 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:16:01.557638 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.557594 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:16:01.557638 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.557608 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:16:01.557988 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.557819 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" (UID: "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:16:01.559534 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.559501 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" (UID: "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:16:01.559637 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.559506 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" (UID: "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:16:01.559944 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.559923 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" (UID: "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:16:01.559944 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.559927 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-kube-api-access-kzjgx" (OuterVolumeSpecName: "kube-api-access-kzjgx") pod "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" (UID: "e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69"). InnerVolumeSpecName "kube-api-access-kzjgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:16:01.563322 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.563302 2578 scope.go:117] "RemoveContainer" containerID="59dda6f7ca462cf6919dcf864cf9aa7b7045375ea6504eb7c9aabf052f0aa854" Sep 06 06:16:01.563667 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:16:01.563644 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59dda6f7ca462cf6919dcf864cf9aa7b7045375ea6504eb7c9aabf052f0aa854\": container with ID starting with 59dda6f7ca462cf6919dcf864cf9aa7b7045375ea6504eb7c9aabf052f0aa854 not found: ID does not exist" containerID="59dda6f7ca462cf6919dcf864cf9aa7b7045375ea6504eb7c9aabf052f0aa854" Sep 06 06:16:01.563745 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.563677 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59dda6f7ca462cf6919dcf864cf9aa7b7045375ea6504eb7c9aabf052f0aa854"} err="failed to get container status \"59dda6f7ca462cf6919dcf864cf9aa7b7045375ea6504eb7c9aabf052f0aa854\": rpc error: code = NotFound desc = could not find container \"59dda6f7ca462cf6919dcf864cf9aa7b7045375ea6504eb7c9aabf052f0aa854\": container with ID starting with 59dda6f7ca462cf6919dcf864cf9aa7b7045375ea6504eb7c9aabf052f0aa854 not found: ID does not exist" Sep 06 06:16:01.563745 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.563702 2578 scope.go:117] "RemoveContainer" containerID="2ad8a914be0d04bde00a5f902e0a7d9f8ac4bfb87159481c66a44f7949cc1402" Sep 06 06:16:01.564057 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:16:01.564032 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad8a914be0d04bde00a5f902e0a7d9f8ac4bfb87159481c66a44f7949cc1402\": container with ID starting with 2ad8a914be0d04bde00a5f902e0a7d9f8ac4bfb87159481c66a44f7949cc1402 not found: ID does not exist" containerID="2ad8a914be0d04bde00a5f902e0a7d9f8ac4bfb87159481c66a44f7949cc1402" Sep 06 06:16:01.564099 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.564064 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad8a914be0d04bde00a5f902e0a7d9f8ac4bfb87159481c66a44f7949cc1402"} err="failed to get container status \"2ad8a914be0d04bde00a5f902e0a7d9f8ac4bfb87159481c66a44f7949cc1402\": rpc error: code = NotFound desc = could not find container \"2ad8a914be0d04bde00a5f902e0a7d9f8ac4bfb87159481c66a44f7949cc1402\": container with ID starting with 2ad8a914be0d04bde00a5f902e0a7d9f8ac4bfb87159481c66a44f7949cc1402 not found: ID does not exist" Sep 06 06:16:01.659077 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.659044 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:16:01.659077 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.659075 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzjgx\" (UniqueName: \"kubernetes.io/projected/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-kube-api-access-kzjgx\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:16:01.659251 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.659091 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:16:01.659251 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.659100 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:16:01.659251 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.659108 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:16:01.869484 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.869449 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb"] Sep 06 06:16:01.873980 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:01.873954 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fc95779d-openshift-default-d5bf587d4-g8bhb"] Sep 06 06:16:03.144564 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:03.144527 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" path="/var/lib/kubelet/pods/e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69/volumes" Sep 06 06:16:09.546392 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546358 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg"] Sep 06 06:16:09.546862 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546719 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452e293c-1514-4cca-b4c8-d0d4e10a0290" containerName="istio-proxy" Sep 06 06:16:09.546862 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546747 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="452e293c-1514-4cca-b4c8-d0d4e10a0290" containerName="istio-proxy" Sep 06 06:16:09.546862 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546762 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerName="istio-proxy" Sep 06 06:16:09.546862 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546768 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerName="istio-proxy" Sep 06 06:16:09.546862 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546775 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebab2f82-2062-46dc-9818-9292a4617b12" containerName="revoke-keys" Sep 06 06:16:09.546862 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546781 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebab2f82-2062-46dc-9818-9292a4617b12" containerName="revoke-keys" Sep 06 06:16:09.546862 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546798 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerName="istio-proxy" Sep 06 06:16:09.546862 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546803 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerName="istio-proxy" Sep 06 06:16:09.546862 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546812 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" containerName="istio-proxy" Sep 06 06:16:09.546862 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546818 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" containerName="istio-proxy" Sep 06 06:16:09.546862 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546828 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" containerName="istio-proxy" Sep 06 06:16:09.546862 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546832 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" containerName="istio-proxy" Sep 06 06:16:09.547533 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546893 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="452e293c-1514-4cca-b4c8-d0d4e10a0290" containerName="istio-proxy" Sep 06 06:16:09.547533 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546914 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerName="istio-proxy" Sep 06 06:16:09.547533 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546920 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" containerName="istio-proxy" Sep 06 06:16:09.547533 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.546927 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebab2f82-2062-46dc-9818-9292a4617b12" containerName="revoke-keys" Sep 06 06:16:09.551504 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.551483 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.553548 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.553526 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-340e00-openshift-default-dockercfg-2qrlt\"" Sep 06 06:16:09.560637 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.560607 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg"] Sep 06 06:16:09.635178 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.635142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-workload-certs\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.635178 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.635179 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-data\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.635401 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.635235 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istiod-ca-cert\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.635401 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.635307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfzrq\" (UniqueName: \"kubernetes.io/projected/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-kube-api-access-sfzrq\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.635401 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.635328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-envoy\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.635401 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.635360 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-workload-socket\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.635535 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.635420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-podinfo\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.635535 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.635446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-token\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.635535 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.635471 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-credential-socket\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.735975 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.735936 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-workload-socket\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.736175 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.735986 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-podinfo\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.736175 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.736006 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-token\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.736175 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.736022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-credential-socket\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.736175 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.736071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-workload-certs\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.736175 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.736086 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-data\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.736175 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.736127 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istiod-ca-cert\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.736175 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.736168 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfzrq\" (UniqueName: \"kubernetes.io/projected/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-kube-api-access-sfzrq\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.736551 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.736197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-envoy\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.736551 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.736370 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-workload-socket\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.736551 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.736385 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-credential-socket\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.736551 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.736449 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-workload-certs\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.736551 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.736503 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-data\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.737089 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.737063 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istiod-ca-cert\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.738807 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.738783 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-podinfo\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.739028 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.739011 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-envoy\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.745842 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.745812 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-token\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.745974 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.745859 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfzrq\" (UniqueName: \"kubernetes.io/projected/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-kube-api-access-sfzrq\") pod \"e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:09.863572 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:09.863490 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:10.136664 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:10.136636 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg"] Sep 06 06:16:10.138613 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:16:10.138585 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72e2000_5d2c_488d_ba88_9e23c8eee0c7.slice/crio-f4edb02c711203115a638900f50ef7c19c553d629bec386a1507cb3cb1f4d40b WatchSource:0}: Error finding container f4edb02c711203115a638900f50ef7c19c553d629bec386a1507cb3cb1f4d40b: Status 404 returned error can't find the container with id f4edb02c711203115a638900f50ef7c19c553d629bec386a1507cb3cb1f4d40b Sep 06 06:16:10.140881 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:10.140854 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:10.140990 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:10.140938 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:10.140990 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:10.140973 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:10.599482 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:10.599441 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" event={"ID":"e72e2000-5d2c-488d-ba88-9e23c8eee0c7","Type":"ContainerStarted","Data":"0c774ee4b3b4ad242006540339b9862938f1791c928426ee5d718af382e4959e"} Sep 06 06:16:10.599864 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:10.599506 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" event={"ID":"e72e2000-5d2c-488d-ba88-9e23c8eee0c7","Type":"ContainerStarted","Data":"f4edb02c711203115a638900f50ef7c19c553d629bec386a1507cb3cb1f4d40b"} Sep 06 06:16:10.619522 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:10.619467 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" podStartSLOduration=1.619448667 podStartE2EDuration="1.619448667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:16:10.616739927 +0000 UTC m=+1336.015070413" watchObservedRunningTime="2026-09-06 06:16:10.619448667 +0000 UTC m=+1336.017779135" Sep 06 06:16:10.864587 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:10.864497 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:11.870361 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:11.870328 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:12.617248 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:12.617215 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:12.618219 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:12.618199 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:17.585324 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:17.585242 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:16:27.619035 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:27.618948 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 06 06:16:27.619035 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:27.619010 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" podUID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 06 06:16:27.711584 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:27.711491 2578 generic.go:356] "Generic (PLEG): container finished" podID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerID="0c774ee4b3b4ad242006540339b9862938f1791c928426ee5d718af382e4959e" exitCode=0 Sep 06 06:16:27.711732 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:27.711569 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" event={"ID":"e72e2000-5d2c-488d-ba88-9e23c8eee0c7","Type":"ContainerDied","Data":"0c774ee4b3b4ad242006540339b9862938f1791c928426ee5d718af382e4959e"} Sep 06 06:16:27.712097 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:27.712081 2578 scope.go:117] "RemoveContainer" containerID="0c774ee4b3b4ad242006540339b9862938f1791c928426ee5d718af382e4959e" Sep 06 06:16:27.713090 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:27.713064 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:27.713151 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:27.713131 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:27.713187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:27.713180 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:28.718575 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:28.718540 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" event={"ID":"e72e2000-5d2c-488d-ba88-9e23c8eee0c7","Type":"ContainerStarted","Data":"20b627f666df6636620ae669816741872e1add453b5b9b0488d96ffa43c1048d"} Sep 06 06:16:28.864204 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:28.864167 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:28.864544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:28.864514 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 06 06:16:28.864746 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:28.864718 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" podUID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 06 06:16:29.864982 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:29.864940 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 06 06:16:29.865482 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:29.865008 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" podUID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 06 06:16:30.864643 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:30.864600 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 06 06:16:30.864856 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:30.864667 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" podUID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 06 06:16:31.864784 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:31.864739 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 06 06:16:31.865333 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:31.864821 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" podUID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 06 06:16:32.864886 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:32.864847 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 06 06:16:32.865479 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:32.865446 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" podUID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 06 06:16:33.864393 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:33.864345 2578 patch_prober.go:28] interesting pod/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 06 06:16:33.864699 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:33.864432 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" podUID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 06 06:16:34.868195 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:34.868163 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:34.868620 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:34.868548 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:34.869392 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:34.869376 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:16:36.216384 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.216339 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7"] Sep 06 06:16:36.216956 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.216937 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452e293c-1514-4cca-b4c8-d0d4e10a0290" containerName="istio-proxy" Sep 06 06:16:36.217047 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.216959 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="452e293c-1514-4cca-b4c8-d0d4e10a0290" containerName="istio-proxy" Sep 06 06:16:36.217114 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.217100 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="452e293c-1514-4cca-b4c8-d0d4e10a0290" containerName="istio-proxy" Sep 06 06:16:36.217176 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.217120 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd5046cd-6f68-45bf-8e6b-59444f9af36d" containerName="istio-proxy" Sep 06 06:16:36.217176 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.217130 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9a5cc1d-3dc7-4e4b-ae50-f05435be5f69" containerName="istio-proxy" Sep 06 06:16:36.220324 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.220302 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.222432 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.222410 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-dc97f9-openshift-default-dockercfg-cdd7l\"" Sep 06 06:16:36.231184 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.231164 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7"] Sep 06 06:16:36.304105 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.304067 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-workload-certs\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.304105 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.304104 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsh29\" (UniqueName: \"kubernetes.io/projected/6f296ad6-71a1-4e88-99bb-62b836296b7d-kube-api-access-fsh29\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.304368 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.304147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f296ad6-71a1-4e88-99bb-62b836296b7d-istiod-ca-cert\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.304368 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.304213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-podinfo\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.304368 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.304251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-data\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.304368 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.304279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-token\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.304368 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.304320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-credential-socket\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.304566 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.304372 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-workload-socket\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.304566 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.304411 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-envoy\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.405397 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.405358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-workload-socket\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.405397 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.405405 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-envoy\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.405651 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.405446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-workload-certs\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.405651 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.405464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsh29\" (UniqueName: \"kubernetes.io/projected/6f296ad6-71a1-4e88-99bb-62b836296b7d-kube-api-access-fsh29\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.405651 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.405506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f296ad6-71a1-4e88-99bb-62b836296b7d-istiod-ca-cert\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.405651 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.405527 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-podinfo\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.405651 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.405555 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-data\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.405976 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.405728 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-token\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.405976 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.405791 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-credential-socket\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.405976 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.405810 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-workload-socket\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.405976 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.405850 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-workload-certs\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.406152 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.405995 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-data\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.406186 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.406170 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-credential-socket\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.406308 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.406286 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f296ad6-71a1-4e88-99bb-62b836296b7d-istiod-ca-cert\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.407990 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.407973 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-envoy\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.408359 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.408339 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-podinfo\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.412515 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.412492 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsh29\" (UniqueName: \"kubernetes.io/projected/6f296ad6-71a1-4e88-99bb-62b836296b7d-kube-api-access-fsh29\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.412620 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.412548 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-token\") pod \"e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.532443 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.532408 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:36.807259 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.807229 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7"] Sep 06 06:16:36.808970 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:16:36.808939 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f296ad6_71a1_4e88_99bb_62b836296b7d.slice/crio-758913b6964646d830763519dbf6708cc6b597543b13985e723cad6c95d88d17 WatchSource:0}: Error finding container 758913b6964646d830763519dbf6708cc6b597543b13985e723cad6c95d88d17: Status 404 returned error can't find the container with id 758913b6964646d830763519dbf6708cc6b597543b13985e723cad6c95d88d17 Sep 06 06:16:36.811569 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.811538 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:36.811683 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.811616 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:36.811683 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:36.811658 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:37.772690 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:37.772655 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" event={"ID":"6f296ad6-71a1-4e88-99bb-62b836296b7d","Type":"ContainerStarted","Data":"0612a1bf22de3c96f48fc79f347062a4d764dd253e19bbc08ab942d433ca1941"} Sep 06 06:16:37.773088 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:37.772718 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" event={"ID":"6f296ad6-71a1-4e88-99bb-62b836296b7d","Type":"ContainerStarted","Data":"758913b6964646d830763519dbf6708cc6b597543b13985e723cad6c95d88d17"} Sep 06 06:16:37.791423 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:37.791373 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" podStartSLOduration=1.791359207 podStartE2EDuration="1.791359207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:16:37.789971925 +0000 UTC m=+1363.188302418" watchObservedRunningTime="2026-09-06 06:16:37.791359207 +0000 UTC m=+1363.189689674" Sep 06 06:16:38.131069 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.130991 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29"] Sep 06 06:16:38.134679 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.134660 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.137062 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.137036 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b7c728-openshift-default-dockercfg-pbc5c\"" Sep 06 06:16:38.146475 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.146453 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29"] Sep 06 06:16:38.223152 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.223115 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms7dc\" (UniqueName: \"kubernetes.io/projected/1d1a22d3-db9b-48ec-bd59-6067b4086dec-kube-api-access-ms7dc\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.223152 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.223154 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-workload-certs\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.223365 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.223265 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-token\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.223365 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.223306 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istiod-ca-cert\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.223365 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.223345 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-envoy\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.223468 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.223367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-data\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.223468 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.223388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-credential-socket\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.223468 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.223421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-workload-socket\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.223468 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.223459 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-podinfo\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.324552 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.324514 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-credential-socket\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.324552 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.324552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-workload-socket\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.324804 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.324590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-podinfo\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.324804 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.324639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms7dc\" (UniqueName: \"kubernetes.io/projected/1d1a22d3-db9b-48ec-bd59-6067b4086dec-kube-api-access-ms7dc\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.324804 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.324660 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-workload-certs\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.324804 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.324717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-token\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.324804 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.324742 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istiod-ca-cert\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.324804 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.324762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-envoy\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.324804 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.324783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-data\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.325232 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.325016 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-credential-socket\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.325232 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.325050 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-workload-socket\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.325232 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.325085 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-workload-certs\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.325232 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.325153 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-data\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.325550 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.325531 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istiod-ca-cert\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.327191 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.327169 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-podinfo\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.327286 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.327203 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-envoy\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.331977 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.331959 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-token\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.332242 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.332226 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms7dc\" (UniqueName: \"kubernetes.io/projected/1d1a22d3-db9b-48ec-bd59-6067b4086dec-kube-api-access-ms7dc\") pod \"e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.449255 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.449163 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:38.533602 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.533572 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:38.538330 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.538302 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:38.726494 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.726385 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29"] Sep 06 06:16:38.728784 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:16:38.728754 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1a22d3_db9b_48ec_bd59_6067b4086dec.slice/crio-daf28eb2b49d5c617cc3b45ad79cc6f49af38bf63a9c9d2099ee9fa9d25cbc2d WatchSource:0}: Error finding container daf28eb2b49d5c617cc3b45ad79cc6f49af38bf63a9c9d2099ee9fa9d25cbc2d: Status 404 returned error can't find the container with id daf28eb2b49d5c617cc3b45ad79cc6f49af38bf63a9c9d2099ee9fa9d25cbc2d Sep 06 06:16:38.731446 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.731414 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:38.731526 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.731477 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:38.731526 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.731508 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:38.780445 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.780413 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" event={"ID":"1d1a22d3-db9b-48ec-bd59-6067b4086dec","Type":"ContainerStarted","Data":"daf28eb2b49d5c617cc3b45ad79cc6f49af38bf63a9c9d2099ee9fa9d25cbc2d"} Sep 06 06:16:38.780800 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.780716 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:38.781670 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:38.781656 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:16:39.797490 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:39.797427 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" event={"ID":"1d1a22d3-db9b-48ec-bd59-6067b4086dec","Type":"ContainerStarted","Data":"148f919d44d8463c66ae8b198af3d13e004f251573f6f1c40336065c82ff4da4"} Sep 06 06:16:40.450088 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:40.450051 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:40.455542 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:40.455512 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:40.475260 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:40.475205 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" podStartSLOduration=2.475186235 podStartE2EDuration="2.475186235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:16:39.819512047 +0000 UTC m=+1365.217842517" watchObservedRunningTime="2026-09-06 06:16:40.475186235 +0000 UTC m=+1365.873516702" Sep 06 06:16:40.803512 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:40.803476 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:40.804603 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:40.804578 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:47.082658 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:47.082621 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:16:56.845097 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:56.845038 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 06 06:16:56.845633 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:56.845104 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" podUID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 06 06:16:57.921325 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:57.921289 2578 generic.go:356] "Generic (PLEG): container finished" podID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" containerID="148f919d44d8463c66ae8b198af3d13e004f251573f6f1c40336065c82ff4da4" exitCode=0 Sep 06 06:16:57.921871 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:57.921366 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" event={"ID":"1d1a22d3-db9b-48ec-bd59-6067b4086dec","Type":"ContainerDied","Data":"148f919d44d8463c66ae8b198af3d13e004f251573f6f1c40336065c82ff4da4"} Sep 06 06:16:57.921871 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:57.921807 2578 scope.go:117] "RemoveContainer" containerID="148f919d44d8463c66ae8b198af3d13e004f251573f6f1c40336065c82ff4da4" Sep 06 06:16:57.923184 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:57.923145 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:57.923285 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:57.923260 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:57.923326 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:57.923305 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:58.928511 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:58.928477 2578 generic.go:356] "Generic (PLEG): container finished" podID="6f296ad6-71a1-4e88-99bb-62b836296b7d" containerID="0612a1bf22de3c96f48fc79f347062a4d764dd253e19bbc08ab942d433ca1941" exitCode=0 Sep 06 06:16:58.929009 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:58.928549 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" event={"ID":"6f296ad6-71a1-4e88-99bb-62b836296b7d","Type":"ContainerDied","Data":"0612a1bf22de3c96f48fc79f347062a4d764dd253e19bbc08ab942d433ca1941"} Sep 06 06:16:58.929086 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:58.929019 2578 scope.go:117] "RemoveContainer" containerID="0612a1bf22de3c96f48fc79f347062a4d764dd253e19bbc08ab942d433ca1941" Sep 06 06:16:58.930176 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:58.930143 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:58.930293 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:58.930225 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:58.930293 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:58.930273 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:16:58.930673 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:58.930653 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" event={"ID":"1d1a22d3-db9b-48ec-bd59-6067b4086dec","Type":"ContainerStarted","Data":"3bfa94432733935b578b82719fb642ef5ba1eb44ddc64166a339ad79c81ffdd7"} Sep 06 06:16:59.451728 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:59.451687 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:16:59.452010 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:59.451975 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 06 06:16:59.452095 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:59.452059 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" podUID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 06 06:16:59.938671 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:16:59.938636 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" event={"ID":"6f296ad6-71a1-4e88-99bb-62b836296b7d","Type":"ContainerStarted","Data":"c60dab4dea7b275d93c937406fdcfe7b0a0a0b2b3516b25b8b14dc723a8274ef"} Sep 06 06:17:00.450582 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:00.450540 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 06 06:17:00.450947 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:00.450877 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" podUID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 06 06:17:00.533186 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:00.533149 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:17:00.533186 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:00.533177 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 06 06:17:00.533400 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:00.533222 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" podUID="6f296ad6-71a1-4e88-99bb-62b836296b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 06 06:17:01.450356 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:01.450318 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 06 06:17:01.451007 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:01.450963 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" podUID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 06 06:17:01.533513 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:01.533478 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 06 06:17:01.533687 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:01.533548 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" podUID="6f296ad6-71a1-4e88-99bb-62b836296b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 06 06:17:02.450421 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:02.450383 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 06 06:17:02.451086 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:02.451051 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" podUID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 06 06:17:02.533550 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:02.533512 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 06 06:17:02.533755 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:02.533572 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" podUID="6f296ad6-71a1-4e88-99bb-62b836296b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 06 06:17:03.450496 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:03.450453 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 06 06:17:03.451135 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:03.451077 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" podUID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 06 06:17:03.533567 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:03.533526 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 06 06:17:03.533744 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:03.533581 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" podUID="6f296ad6-71a1-4e88-99bb-62b836296b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 06 06:17:04.469503 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:04.469469 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:17:04.470762 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:04.470733 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:17:04.476615 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:04.476585 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:17:04.533720 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:04.533687 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Sep 06 06:17:04.533933 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:04.533738 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" podUID="6f296ad6-71a1-4e88-99bb-62b836296b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Sep 06 06:17:05.555518 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:05.555487 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:17:05.556043 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:05.555752 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:17:05.557045 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:05.557024 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:17:07.280091 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:07.280052 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7725-139701829121632"] Sep 06 06:17:07.283794 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:07.283777 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:07.286480 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:07.286455 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-j266t\"" Sep 06 06:17:07.288194 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:07.288170 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-139701829121632"] Sep 06 06:17:07.338221 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:07.338189 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvbxt\" (UniqueName: \"kubernetes.io/projected/f8d0fb37-ecef-450b-952f-0d41923bc085-kube-api-access-kvbxt\") pod \"test-curl-7725-139701829121632\" (UID: \"f8d0fb37-ecef-450b-952f-0d41923bc085\") " pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:07.438719 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:07.438675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvbxt\" (UniqueName: \"kubernetes.io/projected/f8d0fb37-ecef-450b-952f-0d41923bc085-kube-api-access-kvbxt\") pod \"test-curl-7725-139701829121632\" (UID: \"f8d0fb37-ecef-450b-952f-0d41923bc085\") " pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:07.448793 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:07.448758 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvbxt\" (UniqueName: \"kubernetes.io/projected/f8d0fb37-ecef-450b-952f-0d41923bc085-kube-api-access-kvbxt\") pod \"test-curl-7725-139701829121632\" (UID: \"f8d0fb37-ecef-450b-952f-0d41923bc085\") " pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:07.595109 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:07.595014 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:07.864203 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:07.864179 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-139701829121632"] Sep 06 06:17:07.866503 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:17:07.866478 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8d0fb37_ecef_450b_952f_0d41923bc085.slice/crio-094a3523beaff25853855a20d2051af547c7a3c66da0a40fd865fb99e298545f WatchSource:0}: Error finding container 094a3523beaff25853855a20d2051af547c7a3c66da0a40fd865fb99e298545f: Status 404 returned error can't find the container with id 094a3523beaff25853855a20d2051af547c7a3c66da0a40fd865fb99e298545f Sep 06 06:17:07.867754 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:07.867737 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 06:17:08.003018 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:08.002982 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-139701829121632" event={"ID":"f8d0fb37-ecef-450b-952f-0d41923bc085","Type":"ContainerStarted","Data":"094a3523beaff25853855a20d2051af547c7a3c66da0a40fd865fb99e298545f"} Sep 06 06:17:09.012699 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:09.012657 2578 generic.go:356] "Generic (PLEG): container finished" podID="f8d0fb37-ecef-450b-952f-0d41923bc085" containerID="23bef4cdd197a6f9f2449ab60c5c7fa287704b541856726b069faf8d83b8b2dd" exitCode=0 Sep 06 06:17:09.013216 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:09.012733 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-139701829121632" event={"ID":"f8d0fb37-ecef-450b-952f-0d41923bc085","Type":"ContainerDied","Data":"23bef4cdd197a6f9f2449ab60c5c7fa287704b541856726b069faf8d83b8b2dd"} Sep 06 06:17:10.169797 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.169773 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:10.201599 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.201566 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7725-139701829121632_f8d0fb37-ecef-450b-952f-0d41923bc085/test-curl-7725-139701829121632/0.log" Sep 06 06:17:10.221452 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.221406 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7725-139701829121632"] Sep 06 06:17:10.227258 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.227225 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7725-139701829121632"] Sep 06 06:17:10.258848 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.258810 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvbxt\" (UniqueName: \"kubernetes.io/projected/f8d0fb37-ecef-450b-952f-0d41923bc085-kube-api-access-kvbxt\") pod \"f8d0fb37-ecef-450b-952f-0d41923bc085\" (UID: \"f8d0fb37-ecef-450b-952f-0d41923bc085\") " Sep 06 06:17:10.261097 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.261071 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d0fb37-ecef-450b-952f-0d41923bc085-kube-api-access-kvbxt" (OuterVolumeSpecName: "kube-api-access-kvbxt") pod "f8d0fb37-ecef-450b-952f-0d41923bc085" (UID: "f8d0fb37-ecef-450b-952f-0d41923bc085"). InnerVolumeSpecName "kube-api-access-kvbxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:17:10.342577 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.342490 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7725-139701829121632"] Sep 06 06:17:10.342967 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.342952 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d0fb37-ecef-450b-952f-0d41923bc085" containerName="test-curl-7725-139701829121632" Sep 06 06:17:10.343024 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.342969 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d0fb37-ecef-450b-952f-0d41923bc085" containerName="test-curl-7725-139701829121632" Sep 06 06:17:10.343061 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.343041 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d0fb37-ecef-450b-952f-0d41923bc085" containerName="test-curl-7725-139701829121632" Sep 06 06:17:10.346248 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.346228 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:10.351347 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.351319 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-139701829121632"] Sep 06 06:17:10.359728 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.359699 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj7s5\" (UniqueName: \"kubernetes.io/projected/39d4758c-7811-4f35-a314-1fb3d9c57688-kube-api-access-nj7s5\") pod \"test-curl-7725-139701829121632\" (UID: \"39d4758c-7811-4f35-a314-1fb3d9c57688\") " pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:10.359930 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.359885 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvbxt\" (UniqueName: \"kubernetes.io/projected/f8d0fb37-ecef-450b-952f-0d41923bc085-kube-api-access-kvbxt\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:17:10.460759 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.460701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj7s5\" (UniqueName: \"kubernetes.io/projected/39d4758c-7811-4f35-a314-1fb3d9c57688-kube-api-access-nj7s5\") pod \"test-curl-7725-139701829121632\" (UID: \"39d4758c-7811-4f35-a314-1fb3d9c57688\") " pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:10.468509 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.468480 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj7s5\" (UniqueName: \"kubernetes.io/projected/39d4758c-7811-4f35-a314-1fb3d9c57688-kube-api-access-nj7s5\") pod \"test-curl-7725-139701829121632\" (UID: \"39d4758c-7811-4f35-a314-1fb3d9c57688\") " pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:10.658699 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.658595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:10.929525 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:10.929496 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-139701829121632"] Sep 06 06:17:10.931233 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:17:10.931209 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39d4758c_7811_4f35_a314_1fb3d9c57688.slice/crio-c32a6ac24d9c19a49d38cec193700a566feef5c51926b52593352c8ca06eb9a4 WatchSource:0}: Error finding container c32a6ac24d9c19a49d38cec193700a566feef5c51926b52593352c8ca06eb9a4: Status 404 returned error can't find the container with id c32a6ac24d9c19a49d38cec193700a566feef5c51926b52593352c8ca06eb9a4 Sep 06 06:17:11.026667 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:11.026636 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="094a3523beaff25853855a20d2051af547c7a3c66da0a40fd865fb99e298545f" Sep 06 06:17:11.026667 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:11.026646 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:11.027840 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:11.027808 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-139701829121632" event={"ID":"39d4758c-7811-4f35-a314-1fb3d9c57688","Type":"ContainerStarted","Data":"c32a6ac24d9c19a49d38cec193700a566feef5c51926b52593352c8ca06eb9a4"} Sep 06 06:17:11.037988 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:11.037955 2578 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7725-139701829121632" oldPodUID="f8d0fb37-ecef-450b-952f-0d41923bc085" podUID="39d4758c-7811-4f35-a314-1fb3d9c57688" Sep 06 06:17:11.144588 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:11.144549 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8d0fb37-ecef-450b-952f-0d41923bc085" path="/var/lib/kubelet/pods/f8d0fb37-ecef-450b-952f-0d41923bc085/volumes" Sep 06 06:17:12.043121 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:12.043080 2578 generic.go:356] "Generic (PLEG): container finished" podID="39d4758c-7811-4f35-a314-1fb3d9c57688" containerID="7872100d80caf279a03c65268fa0102843bbf622c4deb50ff152f2919190d88c" exitCode=0 Sep 06 06:17:12.043529 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:12.043165 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-139701829121632" event={"ID":"39d4758c-7811-4f35-a314-1fb3d9c57688","Type":"ContainerDied","Data":"7872100d80caf279a03c65268fa0102843bbf622c4deb50ff152f2919190d88c"} Sep 06 06:17:13.190508 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.190482 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:13.208426 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.208393 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7725-139701829121632_39d4758c-7811-4f35-a314-1fb3d9c57688/test-curl-7725-139701829121632/0.log" Sep 06 06:17:13.231556 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.231521 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7725-139701829121632"] Sep 06 06:17:13.235442 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.235417 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7725-139701829121632"] Sep 06 06:17:13.291476 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.291417 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj7s5\" (UniqueName: \"kubernetes.io/projected/39d4758c-7811-4f35-a314-1fb3d9c57688-kube-api-access-nj7s5\") pod \"39d4758c-7811-4f35-a314-1fb3d9c57688\" (UID: \"39d4758c-7811-4f35-a314-1fb3d9c57688\") " Sep 06 06:17:13.294318 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.294287 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d4758c-7811-4f35-a314-1fb3d9c57688-kube-api-access-nj7s5" (OuterVolumeSpecName: "kube-api-access-nj7s5") pod "39d4758c-7811-4f35-a314-1fb3d9c57688" (UID: "39d4758c-7811-4f35-a314-1fb3d9c57688"). InnerVolumeSpecName "kube-api-access-nj7s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:17:13.354250 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.354213 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7725-139701829122208"] Sep 06 06:17:13.354842 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.354823 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d4758c-7811-4f35-a314-1fb3d9c57688" containerName="test-curl-7725-139701829121632" Sep 06 06:17:13.354887 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.354847 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d4758c-7811-4f35-a314-1fb3d9c57688" containerName="test-curl-7725-139701829121632" Sep 06 06:17:13.354984 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.354971 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d4758c-7811-4f35-a314-1fb3d9c57688" containerName="test-curl-7725-139701829121632" Sep 06 06:17:13.358300 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.358281 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:13.361708 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.361675 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-139701829122208"] Sep 06 06:17:13.392697 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.392661 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjmxp\" (UniqueName: \"kubernetes.io/projected/abd97342-83a3-472a-9e7c-49d7f44cacc0-kube-api-access-cjmxp\") pod \"test-curl-7725-139701829122208\" (UID: \"abd97342-83a3-472a-9e7c-49d7f44cacc0\") " pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:13.392867 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.392837 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj7s5\" (UniqueName: \"kubernetes.io/projected/39d4758c-7811-4f35-a314-1fb3d9c57688-kube-api-access-nj7s5\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:17:13.493381 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.493346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjmxp\" (UniqueName: \"kubernetes.io/projected/abd97342-83a3-472a-9e7c-49d7f44cacc0-kube-api-access-cjmxp\") pod \"test-curl-7725-139701829122208\" (UID: \"abd97342-83a3-472a-9e7c-49d7f44cacc0\") " pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:13.500695 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.500667 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjmxp\" (UniqueName: \"kubernetes.io/projected/abd97342-83a3-472a-9e7c-49d7f44cacc0-kube-api-access-cjmxp\") pod \"test-curl-7725-139701829122208\" (UID: \"abd97342-83a3-472a-9e7c-49d7f44cacc0\") " pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:13.668655 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.668570 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:13.953784 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:13.953757 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-139701829122208"] Sep 06 06:17:13.955976 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:17:13.955944 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabd97342_83a3_472a_9e7c_49d7f44cacc0.slice/crio-8bdcf52e7205708df357ec9219398becbfaffd8552a465c40015b550ff7c78d6 WatchSource:0}: Error finding container 8bdcf52e7205708df357ec9219398becbfaffd8552a465c40015b550ff7c78d6: Status 404 returned error can't find the container with id 8bdcf52e7205708df357ec9219398becbfaffd8552a465c40015b550ff7c78d6 Sep 06 06:17:14.055694 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:14.055664 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:14.055694 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:14.055688 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c32a6ac24d9c19a49d38cec193700a566feef5c51926b52593352c8ca06eb9a4" Sep 06 06:17:14.057246 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:14.057210 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-139701829122208" event={"ID":"abd97342-83a3-472a-9e7c-49d7f44cacc0","Type":"ContainerStarted","Data":"8bdcf52e7205708df357ec9219398becbfaffd8552a465c40015b550ff7c78d6"} Sep 06 06:17:14.065963 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:17:14.065936 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7725-139701829121632\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="39d4758c-7811-4f35-a314-1fb3d9c57688" pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:15.065088 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:15.065052 2578 generic.go:356] "Generic (PLEG): container finished" podID="abd97342-83a3-472a-9e7c-49d7f44cacc0" containerID="2a377082283b2c666eb9335801fde1ec87eb920e2ca018ab700d484ed6869648" exitCode=0 Sep 06 06:17:15.065484 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:15.065139 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-139701829122208" event={"ID":"abd97342-83a3-472a-9e7c-49d7f44cacc0","Type":"ContainerDied","Data":"2a377082283b2c666eb9335801fde1ec87eb920e2ca018ab700d484ed6869648"} Sep 06 06:17:15.078062 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:17:15.078033 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7725-139701829121632\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="39d4758c-7811-4f35-a314-1fb3d9c57688" pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:15.144194 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:17:15.144162 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7725-139701829121632\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="39d4758c-7811-4f35-a314-1fb3d9c57688" pod="opendatahub/test-curl-7725-139701829121632" Sep 06 06:17:15.145790 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:15.145769 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39d4758c-7811-4f35-a314-1fb3d9c57688" path="/var/lib/kubelet/pods/39d4758c-7811-4f35-a314-1fb3d9c57688/volumes" Sep 06 06:17:16.209983 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.209956 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:16.228195 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.228166 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7725-139701829122208_abd97342-83a3-472a-9e7c-49d7f44cacc0/test-curl-7725-139701829122208/0.log" Sep 06 06:17:16.248870 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.248834 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7725-139701829122208"] Sep 06 06:17:16.252402 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.252380 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7725-139701829122208"] Sep 06 06:17:16.318812 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.318776 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjmxp\" (UniqueName: \"kubernetes.io/projected/abd97342-83a3-472a-9e7c-49d7f44cacc0-kube-api-access-cjmxp\") pod \"abd97342-83a3-472a-9e7c-49d7f44cacc0\" (UID: \"abd97342-83a3-472a-9e7c-49d7f44cacc0\") " Sep 06 06:17:16.321023 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.320974 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd97342-83a3-472a-9e7c-49d7f44cacc0-kube-api-access-cjmxp" (OuterVolumeSpecName: "kube-api-access-cjmxp") pod "abd97342-83a3-472a-9e7c-49d7f44cacc0" (UID: "abd97342-83a3-472a-9e7c-49d7f44cacc0"). InnerVolumeSpecName "kube-api-access-cjmxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:17:16.365846 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.365814 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7725-139701829122208"] Sep 06 06:17:16.366264 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.366250 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abd97342-83a3-472a-9e7c-49d7f44cacc0" containerName="test-curl-7725-139701829122208" Sep 06 06:17:16.366312 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.366266 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="abd97342-83a3-472a-9e7c-49d7f44cacc0" containerName="test-curl-7725-139701829122208" Sep 06 06:17:16.366347 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.366338 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="abd97342-83a3-472a-9e7c-49d7f44cacc0" containerName="test-curl-7725-139701829122208" Sep 06 06:17:16.369317 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.369302 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:16.375322 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.375297 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-139701829122208"] Sep 06 06:17:16.420021 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.419985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr2pw\" (UniqueName: \"kubernetes.io/projected/cab5ff6a-b51c-4f02-8b72-0a59f1f138df-kube-api-access-sr2pw\") pod \"test-curl-7725-139701829122208\" (UID: \"cab5ff6a-b51c-4f02-8b72-0a59f1f138df\") " pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:16.420173 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.420146 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjmxp\" (UniqueName: \"kubernetes.io/projected/abd97342-83a3-472a-9e7c-49d7f44cacc0-kube-api-access-cjmxp\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:17:16.521287 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.521253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr2pw\" (UniqueName: \"kubernetes.io/projected/cab5ff6a-b51c-4f02-8b72-0a59f1f138df-kube-api-access-sr2pw\") pod \"test-curl-7725-139701829122208\" (UID: \"cab5ff6a-b51c-4f02-8b72-0a59f1f138df\") " pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:16.529684 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.529658 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr2pw\" (UniqueName: \"kubernetes.io/projected/cab5ff6a-b51c-4f02-8b72-0a59f1f138df-kube-api-access-sr2pw\") pod \"test-curl-7725-139701829122208\" (UID: \"cab5ff6a-b51c-4f02-8b72-0a59f1f138df\") " pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:16.680328 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.680242 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:16.949313 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:16.949287 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-139701829122208"] Sep 06 06:17:16.951515 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:17:16.951489 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcab5ff6a_b51c_4f02_8b72_0a59f1f138df.slice/crio-113c0bd7aff8256153683f2146ef682749699a0c912bfb547e607b7db55a88a9 WatchSource:0}: Error finding container 113c0bd7aff8256153683f2146ef682749699a0c912bfb547e607b7db55a88a9: Status 404 returned error can't find the container with id 113c0bd7aff8256153683f2146ef682749699a0c912bfb547e607b7db55a88a9 Sep 06 06:17:17.077792 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:17.077759 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:17.077792 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:17.077796 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bdcf52e7205708df357ec9219398becbfaffd8552a465c40015b550ff7c78d6" Sep 06 06:17:17.079208 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:17.079178 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-139701829122208" event={"ID":"cab5ff6a-b51c-4f02-8b72-0a59f1f138df","Type":"ContainerStarted","Data":"113c0bd7aff8256153683f2146ef682749699a0c912bfb547e607b7db55a88a9"} Sep 06 06:17:17.089264 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:17.089241 2578 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7725-139701829122208" oldPodUID="abd97342-83a3-472a-9e7c-49d7f44cacc0" podUID="cab5ff6a-b51c-4f02-8b72-0a59f1f138df" Sep 06 06:17:17.149021 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:17.148985 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abd97342-83a3-472a-9e7c-49d7f44cacc0" path="/var/lib/kubelet/pods/abd97342-83a3-472a-9e7c-49d7f44cacc0/volumes" Sep 06 06:17:17.685786 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:17.685705 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:17:18.090283 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:18.090247 2578 generic.go:356] "Generic (PLEG): container finished" podID="cab5ff6a-b51c-4f02-8b72-0a59f1f138df" containerID="b16698eeb6c0825d8d1bcc6c0b1d09f6895ce1f8e53e95249e07da21b9c12f67" exitCode=0 Sep 06 06:17:18.090437 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:18.090336 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-139701829122208" event={"ID":"cab5ff6a-b51c-4f02-8b72-0a59f1f138df","Type":"ContainerDied","Data":"b16698eeb6c0825d8d1bcc6c0b1d09f6895ce1f8e53e95249e07da21b9c12f67"} Sep 06 06:17:19.229868 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:19.229839 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:19.247097 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:19.247056 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7725-139701829122208_cab5ff6a-b51c-4f02-8b72-0a59f1f138df/test-curl-7725-139701829122208/0.log" Sep 06 06:17:19.267799 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:19.267743 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7725-139701829122208"] Sep 06 06:17:19.270031 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:19.270009 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7725-139701829122208"] Sep 06 06:17:19.353573 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:19.353502 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sr2pw\" (UniqueName: \"kubernetes.io/projected/cab5ff6a-b51c-4f02-8b72-0a59f1f138df-kube-api-access-sr2pw\") pod \"cab5ff6a-b51c-4f02-8b72-0a59f1f138df\" (UID: \"cab5ff6a-b51c-4f02-8b72-0a59f1f138df\") " Sep 06 06:17:19.355989 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:19.355963 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab5ff6a-b51c-4f02-8b72-0a59f1f138df-kube-api-access-sr2pw" (OuterVolumeSpecName: "kube-api-access-sr2pw") pod "cab5ff6a-b51c-4f02-8b72-0a59f1f138df" (UID: "cab5ff6a-b51c-4f02-8b72-0a59f1f138df"). InnerVolumeSpecName "kube-api-access-sr2pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:17:19.455132 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:19.455101 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr2pw\" (UniqueName: \"kubernetes.io/projected/cab5ff6a-b51c-4f02-8b72-0a59f1f138df-kube-api-access-sr2pw\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:17:20.105560 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.105527 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="113c0bd7aff8256153683f2146ef682749699a0c912bfb547e607b7db55a88a9" Sep 06 06:17:20.105560 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.105527 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:20.115702 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:17:20.115670 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7725-139701829122208\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="cab5ff6a-b51c-4f02-8b72-0a59f1f138df" pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:20.117702 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.117678 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg"] Sep 06 06:17:20.118094 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.118079 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cab5ff6a-b51c-4f02-8b72-0a59f1f138df" containerName="test-curl-7725-139701829122208" Sep 06 06:17:20.118180 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.118095 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cab5ff6a-b51c-4f02-8b72-0a59f1f138df" containerName="test-curl-7725-139701829122208" Sep 06 06:17:20.118180 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.118167 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cab5ff6a-b51c-4f02-8b72-0a59f1f138df" containerName="test-curl-7725-139701829122208" Sep 06 06:17:20.121277 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.121258 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.123164 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:17:20.123139 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7725-139701829122208\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="cab5ff6a-b51c-4f02-8b72-0a59f1f138df" pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:20.123563 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.123549 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-dd8c231a-openshift-default-dockercfg-mbws8\"" Sep 06 06:17:20.130598 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.130578 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg"] Sep 06 06:17:20.163076 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.163044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-data\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.163076 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.163081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-workload-socket\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.163277 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.163119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-podinfo\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.163277 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.163251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-credential-socket\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.163406 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.163319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-workload-certs\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.163406 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.163356 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-envoy\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.163479 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.163435 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-token\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.163518 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.163483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5568fb6-d967-427d-9d42-26b57b1b05e9-istiod-ca-cert\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.163565 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.163528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whjzx\" (UniqueName: \"kubernetes.io/projected/a5568fb6-d967-427d-9d42-26b57b1b05e9-kube-api-access-whjzx\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.264065 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.264025 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-credential-socket\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.264535 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.264084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-workload-certs\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.264535 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.264120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-envoy\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.264535 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.264158 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-token\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.264535 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.264296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5568fb6-d967-427d-9d42-26b57b1b05e9-istiod-ca-cert\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.264535 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.264353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whjzx\" (UniqueName: \"kubernetes.io/projected/a5568fb6-d967-427d-9d42-26b57b1b05e9-kube-api-access-whjzx\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.264535 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.264402 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-credential-socket\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.264535 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.264413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-data\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.264535 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.264462 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-workload-certs\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.264535 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.264467 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-workload-socket\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.265010 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.264559 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-podinfo\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.265010 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.264663 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-data\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.265010 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.264866 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-workload-socket\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.265010 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.264994 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5568fb6-d967-427d-9d42-26b57b1b05e9-istiod-ca-cert\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.266596 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.266575 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-envoy\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.266926 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.266888 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-podinfo\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.275986 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.275965 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-token\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.276092 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.276075 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whjzx\" (UniqueName: \"kubernetes.io/projected/a5568fb6-d967-427d-9d42-26b57b1b05e9-kube-api-access-whjzx\") pod \"e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.432317 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.432216 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:20.709296 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.709212 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg"] Sep 06 06:17:20.711457 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:17:20.711436 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5568fb6_d967_427d_9d42_26b57b1b05e9.slice/crio-322e4fdb06df8b7896750b7f062e82bdfd0a2aa908de7f71f8b6c4ff934d72b5 WatchSource:0}: Error finding container 322e4fdb06df8b7896750b7f062e82bdfd0a2aa908de7f71f8b6c4ff934d72b5: Status 404 returned error can't find the container with id 322e4fdb06df8b7896750b7f062e82bdfd0a2aa908de7f71f8b6c4ff934d72b5 Sep 06 06:17:20.713811 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.713782 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:17:20.713883 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.713848 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:17:20.713883 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:20.713873 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:17:21.113624 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:21.113588 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" event={"ID":"a5568fb6-d967-427d-9d42-26b57b1b05e9","Type":"ContainerStarted","Data":"66fc83fa2427833841781535e143302383478bb8b4d72a8a2490f4f3d69d4ea1"} Sep 06 06:17:21.113804 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:21.113640 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" event={"ID":"a5568fb6-d967-427d-9d42-26b57b1b05e9","Type":"ContainerStarted","Data":"322e4fdb06df8b7896750b7f062e82bdfd0a2aa908de7f71f8b6c4ff934d72b5"} Sep 06 06:17:21.115503 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:17:21.115477 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7725-139701829122208\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="cab5ff6a-b51c-4f02-8b72-0a59f1f138df" pod="opendatahub/test-curl-7725-139701829122208" Sep 06 06:17:21.133497 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:21.133449 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" podStartSLOduration=1.13343481 podStartE2EDuration="1.13343481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:17:21.131377619 +0000 UTC m=+1406.529708086" watchObservedRunningTime="2026-09-06 06:17:21.13343481 +0000 UTC m=+1406.531765283" Sep 06 06:17:21.146179 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:21.146149 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cab5ff6a-b51c-4f02-8b72-0a59f1f138df" path="/var/lib/kubelet/pods/cab5ff6a-b51c-4f02-8b72-0a59f1f138df/volumes" Sep 06 06:17:21.433032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:21.432945 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:21.438151 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:21.438122 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:22.120051 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:22.120021 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:22.120966 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:22.120949 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:33.282223 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:33.282187 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:17:43.299120 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.299079 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp"] Sep 06 06:17:43.304168 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.304142 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.307345 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.307309 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dc97f9\"/\"default-dockercfg-wj7sx\"" Sep 06 06:17:43.307345 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.307321 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dc97f9\"/\"auth-test-model-dc97f9-kserve-self-signed-certs\"" Sep 06 06:17:43.307543 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.307311 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dc97f9\"/\"kube-root-ca.crt\"" Sep 06 06:17:43.307543 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.307314 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dc97f9\"/\"openshift-service-ca.crt\"" Sep 06 06:17:43.313093 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.313061 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp"] Sep 06 06:17:43.415963 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.415926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd7zj\" (UniqueName: \"kubernetes.io/projected/0425d06d-2049-4114-bdd2-c9328ad5aa97-kube-api-access-cd7zj\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.416247 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.416217 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-tmp-dir\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.416398 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.416330 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-dshm\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.416476 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.416406 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0425d06d-2049-4114-bdd2-c9328ad5aa97-tls-certs\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.416751 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.416606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-model-cache\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.416751 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.416665 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-home\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.519461 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.517345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-dshm\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.519461 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.517397 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0425d06d-2049-4114-bdd2-c9328ad5aa97-tls-certs\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.519461 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.517456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-model-cache\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.519461 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.517487 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-home\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.519461 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.517558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd7zj\" (UniqueName: \"kubernetes.io/projected/0425d06d-2049-4114-bdd2-c9328ad5aa97-kube-api-access-cd7zj\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.519461 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.517585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-tmp-dir\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.519461 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.517968 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-model-cache\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.519461 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.518014 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-tmp-dir\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.519461 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.518241 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-home\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.520202 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.520171 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-dshm\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.520325 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.520245 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0425d06d-2049-4114-bdd2-c9328ad5aa97-tls-certs\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.525272 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.525243 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd7zj\" (UniqueName: \"kubernetes.io/projected/0425d06d-2049-4114-bdd2-c9328ad5aa97-kube-api-access-cd7zj\") pod \"auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.617176 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.617088 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:43.887497 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:43.887419 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp"] Sep 06 06:17:43.891256 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:17:43.891227 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0425d06d_2049_4114_bdd2_c9328ad5aa97.slice/crio-510f7d167b4e5ccd787e934fa2e2c3a4b1764d60ba4ef6f2bbbfda699bbb8d0f WatchSource:0}: Error finding container 510f7d167b4e5ccd787e934fa2e2c3a4b1764d60ba4ef6f2bbbfda699bbb8d0f: Status 404 returned error can't find the container with id 510f7d167b4e5ccd787e934fa2e2c3a4b1764d60ba4ef6f2bbbfda699bbb8d0f Sep 06 06:17:44.258069 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:44.257984 2578 generic.go:356] "Generic (PLEG): container finished" podID="a5568fb6-d967-427d-9d42-26b57b1b05e9" containerID="66fc83fa2427833841781535e143302383478bb8b4d72a8a2490f4f3d69d4ea1" exitCode=0 Sep 06 06:17:44.258226 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:44.258059 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" event={"ID":"a5568fb6-d967-427d-9d42-26b57b1b05e9","Type":"ContainerDied","Data":"66fc83fa2427833841781535e143302383478bb8b4d72a8a2490f4f3d69d4ea1"} Sep 06 06:17:44.258598 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:44.258582 2578 scope.go:117] "RemoveContainer" containerID="66fc83fa2427833841781535e143302383478bb8b4d72a8a2490f4f3d69d4ea1" Sep 06 06:17:44.259828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:44.259794 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:17:44.259967 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:44.259830 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" event={"ID":"0425d06d-2049-4114-bdd2-c9328ad5aa97","Type":"ContainerStarted","Data":"da43980fddf166b19a772ad4a95c95d54b11ccbacfd5c84c019a5266dfbea880"} Sep 06 06:17:44.259967 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:44.259857 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" event={"ID":"0425d06d-2049-4114-bdd2-c9328ad5aa97","Type":"ContainerStarted","Data":"510f7d167b4e5ccd787e934fa2e2c3a4b1764d60ba4ef6f2bbbfda699bbb8d0f"} Sep 06 06:17:44.259967 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:44.259896 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:17:44.259967 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:44.259954 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:17:44.293935 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:44.293862 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" podStartSLOduration=1.293846248 podStartE2EDuration="1.293846248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:17:44.291844456 +0000 UTC m=+1429.690174961" watchObservedRunningTime="2026-09-06 06:17:44.293846248 +0000 UTC m=+1429.692176716" Sep 06 06:17:45.272559 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:45.272515 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" event={"ID":"a5568fb6-d967-427d-9d42-26b57b1b05e9","Type":"ContainerStarted","Data":"3e87e3aea6115ebf339ee628594987c25a3eb7f8dbe24184874df111db8b3195"} Sep 06 06:17:45.433559 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:45.433131 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:45.433559 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:45.433386 2578 patch_prober.go:28] interesting pod/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 06 06:17:45.433559 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:45.433464 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" podUID="a5568fb6-d967-427d-9d42-26b57b1b05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 06 06:17:46.432984 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:46.432866 2578 patch_prober.go:28] interesting pod/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 06 06:17:46.433359 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:46.433098 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" podUID="a5568fb6-d967-427d-9d42-26b57b1b05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 06 06:17:47.433300 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:47.433262 2578 patch_prober.go:28] interesting pod/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 06 06:17:47.433711 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:47.433320 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" podUID="a5568fb6-d967-427d-9d42-26b57b1b05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 06 06:17:48.433326 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:48.433281 2578 patch_prober.go:28] interesting pod/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 06 06:17:48.433743 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:48.433343 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" podUID="a5568fb6-d967-427d-9d42-26b57b1b05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 06 06:17:49.432733 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:49.432697 2578 patch_prober.go:28] interesting pod/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 06 06:17:49.432939 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:49.432769 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" podUID="a5568fb6-d967-427d-9d42-26b57b1b05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 06 06:17:50.433232 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:50.433194 2578 patch_prober.go:28] interesting pod/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Sep 06 06:17:50.433651 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:50.433256 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" podUID="a5568fb6-d967-427d-9d42-26b57b1b05e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Sep 06 06:17:51.190044 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:51.190006 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:17:51.436971 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:51.436937 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:51.437413 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:51.437214 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:51.437963 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:51.437946 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:17:53.617703 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:53.617668 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:53.618222 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:53.617726 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:53.630280 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:53.630258 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:54.354473 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:54.354444 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:17:55.290153 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:55.290120 2578 scope.go:117] "RemoveContainer" containerID="aed6affe9678a6ee77f9f465c0f3f19a5d34b059fefa3e9fc2c4beac0bdc8d01" Sep 06 06:17:55.299300 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:55.299280 2578 scope.go:117] "RemoveContainer" containerID="f704e5517983739d17c198f48c8b012ffd493d8a529c959684a4ae0195d51cc1" Sep 06 06:17:55.307588 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:17:55.307568 2578 scope.go:117] "RemoveContainer" containerID="e8f6cbadfec095425411d7d6c8f867904b7cf83f889c61cdd1f9aec0c0ff345b" Sep 06 06:18:14.580402 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:14.580362 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:18:29.390328 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:29.390294 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:18:49.286798 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:49.286763 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:18:52.719553 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:52.719516 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7"] Sep 06 06:18:52.723328 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:52.723310 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7" Sep 06 06:18:52.725259 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:52.725234 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:18:52.725259 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:52.725250 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:18:52.729090 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:52.729061 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7"] Sep 06 06:18:52.916795 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:52.916755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d42b8a48-22e6-4e0a-9d87-e21379cb08af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7\" (UID: \"d42b8a48-22e6-4e0a-9d87-e21379cb08af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7" Sep 06 06:18:53.018400 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:53.018375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d42b8a48-22e6-4e0a-9d87-e21379cb08af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7\" (UID: \"d42b8a48-22e6-4e0a-9d87-e21379cb08af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7" Sep 06 06:18:53.019031 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:53.019012 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d42b8a48-22e6-4e0a-9d87-e21379cb08af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7\" (UID: \"d42b8a48-22e6-4e0a-9d87-e21379cb08af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7" Sep 06 06:18:53.036279 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:53.036258 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7" Sep 06 06:18:53.307925 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:53.307801 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7"] Sep 06 06:18:53.311111 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:18:53.311084 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42b8a48_22e6_4e0a_9d87_e21379cb08af.slice/crio-eac94acdc22794076458231b5b0cca312792081a69b4cafd869d687f08f1cc9f WatchSource:0}: Error finding container eac94acdc22794076458231b5b0cca312792081a69b4cafd869d687f08f1cc9f: Status 404 returned error can't find the container with id eac94acdc22794076458231b5b0cca312792081a69b4cafd869d687f08f1cc9f Sep 06 06:18:53.692124 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:53.692094 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7" event={"ID":"d42b8a48-22e6-4e0a-9d87-e21379cb08af","Type":"ContainerStarted","Data":"93b0c36699a422e4436a3b445b4cbf2780d6fb4953ee5de4935c474ab969c276"} Sep 06 06:18:53.692268 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:53.692142 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7" event={"ID":"d42b8a48-22e6-4e0a-9d87-e21379cb08af","Type":"ContainerStarted","Data":"eac94acdc22794076458231b5b0cca312792081a69b4cafd869d687f08f1cc9f"} Sep 06 06:18:53.706513 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:53.706464 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7" podStartSLOduration=1.706433955 podStartE2EDuration="1.706433955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:18:53.70398778 +0000 UTC m=+1499.102318248" watchObservedRunningTime="2026-09-06 06:18:53.706433955 +0000 UTC m=+1499.104764423" Sep 06 06:18:54.700446 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:54.700408 2578 generic.go:356] "Generic (PLEG): container finished" podID="d42b8a48-22e6-4e0a-9d87-e21379cb08af" containerID="93b0c36699a422e4436a3b445b4cbf2780d6fb4953ee5de4935c474ab969c276" exitCode=0 Sep 06 06:18:54.700923 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:54.700495 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7" event={"ID":"d42b8a48-22e6-4e0a-9d87-e21379cb08af","Type":"ContainerDied","Data":"93b0c36699a422e4436a3b445b4cbf2780d6fb4953ee5de4935c474ab969c276"} Sep 06 06:18:55.165050 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:55.165019 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 06:18:55.166828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:55.166804 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 06:18:55.337232 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:55.337200 2578 scope.go:117] "RemoveContainer" containerID="19b02b83f874dc890b07d2319d430db4f52b579a0b4b1014ba037f46b1e257dd" Sep 06 06:18:55.346335 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:55.346315 2578 scope.go:117] "RemoveContainer" containerID="35adf8f19fdb7358ea3c278a0704502af3fcda35e4c456f8a4190c83c756bf9d" Sep 06 06:18:55.844685 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:55.844661 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7" Sep 06 06:18:55.846383 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:55.846364 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d42b8a48-22e6-4e0a-9d87-e21379cb08af-maas-api-service-ca\") pod \"d42b8a48-22e6-4e0a-9d87-e21379cb08af\" (UID: \"d42b8a48-22e6-4e0a-9d87-e21379cb08af\") " Sep 06 06:18:55.846681 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:55.846657 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42b8a48-22e6-4e0a-9d87-e21379cb08af-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d42b8a48-22e6-4e0a-9d87-e21379cb08af" (UID: "d42b8a48-22e6-4e0a-9d87-e21379cb08af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:18:55.947793 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:55.947755 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d42b8a48-22e6-4e0a-9d87-e21379cb08af-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:18:56.713859 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:56.713821 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7" event={"ID":"d42b8a48-22e6-4e0a-9d87-e21379cb08af","Type":"ContainerDied","Data":"eac94acdc22794076458231b5b0cca312792081a69b4cafd869d687f08f1cc9f"} Sep 06 06:18:56.713859 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:56.713861 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7" Sep 06 06:18:56.714110 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:18:56.713882 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eac94acdc22794076458231b5b0cca312792081a69b4cafd869d687f08f1cc9f" Sep 06 06:19:08.387879 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:08.387836 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:19:08.527235 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:08.527197 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7"] Sep 06 06:19:08.529742 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:08.529719 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dd8c231a-qk2q7"] Sep 06 06:19:09.145965 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:09.145927 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d42b8a48-22e6-4e0a-9d87-e21379cb08af" path="/var/lib/kubelet/pods/d42b8a48-22e6-4e0a-9d87-e21379cb08af/volumes" Sep 06 06:19:26.773233 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:26.773152 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:19:43.273362 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:43.273328 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:19:54.644632 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:54.644595 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7728-e83b72"] Sep 06 06:19:54.645043 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:54.645009 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d42b8a48-22e6-4e0a-9d87-e21379cb08af" containerName="revoke-keys" Sep 06 06:19:54.645043 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:54.645023 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d42b8a48-22e6-4e0a-9d87-e21379cb08af" containerName="revoke-keys" Sep 06 06:19:54.645118 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:54.645087 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d42b8a48-22e6-4e0a-9d87-e21379cb08af" containerName="revoke-keys" Sep 06 06:19:54.648105 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:54.648089 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7728-e83b72" Sep 06 06:19:54.650183 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:54.650153 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-j266t\"" Sep 06 06:19:54.651790 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:54.651762 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7728-e83b72"] Sep 06 06:19:54.736323 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:54.736295 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb7zm\" (UniqueName: \"kubernetes.io/projected/a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec-kube-api-access-wb7zm\") pod \"mt-curl-7728-e83b72\" (UID: \"a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec\") " pod="opendatahub/mt-curl-7728-e83b72" Sep 06 06:19:54.837474 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:54.837439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7zm\" (UniqueName: \"kubernetes.io/projected/a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec-kube-api-access-wb7zm\") pod \"mt-curl-7728-e83b72\" (UID: \"a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec\") " pod="opendatahub/mt-curl-7728-e83b72" Sep 06 06:19:54.845931 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:54.845881 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7zm\" (UniqueName: \"kubernetes.io/projected/a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec-kube-api-access-wb7zm\") pod \"mt-curl-7728-e83b72\" (UID: \"a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec\") " pod="opendatahub/mt-curl-7728-e83b72" Sep 06 06:19:54.960433 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:54.960346 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7728-e83b72" Sep 06 06:19:55.231042 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:55.231014 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7728-e83b72"] Sep 06 06:19:55.233108 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:19:55.233085 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda94ba603_9357_4dcf_a1c7_7a6eb0c8cbec.slice/crio-102e96174e5c62544d0dcbe17aedcd09b376989798f444ae82bf8f80fab5105b WatchSource:0}: Error finding container 102e96174e5c62544d0dcbe17aedcd09b376989798f444ae82bf8f80fab5105b: Status 404 returned error can't find the container with id 102e96174e5c62544d0dcbe17aedcd09b376989798f444ae82bf8f80fab5105b Sep 06 06:19:55.371992 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:55.371962 2578 scope.go:117] "RemoveContainer" containerID="89c64ed0b8f7004278e804664a9b120cd4a7142d80cd9c424ebb71b0a76a7b2a" Sep 06 06:19:55.381156 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:55.381135 2578 scope.go:117] "RemoveContainer" containerID="96b44c9b8c626f2de63556aa734ef51b304de546948282702d63f25cad39eef9" Sep 06 06:19:56.058566 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:56.058537 2578 generic.go:356] "Generic (PLEG): container finished" podID="a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec" containerID="d58e901fb8926455c31e02712ad460346a922f436adfcdd4b80f28cf21d82d9d" exitCode=0 Sep 06 06:19:56.058875 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:56.058627 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7728-e83b72" event={"ID":"a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec","Type":"ContainerDied","Data":"d58e901fb8926455c31e02712ad460346a922f436adfcdd4b80f28cf21d82d9d"} Sep 06 06:19:56.058875 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:56.058670 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7728-e83b72" event={"ID":"a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec","Type":"ContainerStarted","Data":"102e96174e5c62544d0dcbe17aedcd09b376989798f444ae82bf8f80fab5105b"} Sep 06 06:19:57.213534 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:57.213511 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7728-e83b72" Sep 06 06:19:57.244839 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:57.244815 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7728-e83b72_a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec/mt-curl-7728-e83b72/0.log" Sep 06 06:19:57.268365 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:57.268328 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7728-e83b72"] Sep 06 06:19:57.270236 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:57.270216 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7728-e83b72"] Sep 06 06:19:57.362359 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:57.362264 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb7zm\" (UniqueName: \"kubernetes.io/projected/a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec-kube-api-access-wb7zm\") pod \"a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec\" (UID: \"a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec\") " Sep 06 06:19:57.364524 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:57.364487 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec-kube-api-access-wb7zm" (OuterVolumeSpecName: "kube-api-access-wb7zm") pod "a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec" (UID: "a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec"). InnerVolumeSpecName "kube-api-access-wb7zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:19:57.463210 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:57.463167 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb7zm\" (UniqueName: \"kubernetes.io/projected/a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec-kube-api-access-wb7zm\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:19:58.071393 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:58.071365 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="102e96174e5c62544d0dcbe17aedcd09b376989798f444ae82bf8f80fab5105b" Sep 06 06:19:58.071576 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:58.071397 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7728-e83b72" Sep 06 06:19:58.080802 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:19:58.080773 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7728-e83b72\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec" pod="opendatahub/mt-curl-7728-e83b72" Sep 06 06:19:58.579018 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:58.578985 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:19:59.145488 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:59.145457 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec" path="/var/lib/kubelet/pods/a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec/volumes" Sep 06 06:19:59.444696 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:59.444598 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg"] Sep 06 06:19:59.444884 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:59.444828 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" podUID="a5568fb6-d967-427d-9d42-26b57b1b05e9" containerName="istio-proxy" containerID="cri-o://3e87e3aea6115ebf339ee628594987c25a3eb7f8dbe24184874df111db8b3195" gracePeriod=30 Sep 06 06:19:59.598141 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:19:59.598105 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp"] Sep 06 06:20:04.707414 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.707382 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:20:04.844213 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.844174 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5568fb6-d967-427d-9d42-26b57b1b05e9-istiod-ca-cert\") pod \"a5568fb6-d967-427d-9d42-26b57b1b05e9\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " Sep 06 06:20:04.844449 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.844255 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whjzx\" (UniqueName: \"kubernetes.io/projected/a5568fb6-d967-427d-9d42-26b57b1b05e9-kube-api-access-whjzx\") pod \"a5568fb6-d967-427d-9d42-26b57b1b05e9\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " Sep 06 06:20:04.844449 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.844277 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-podinfo\") pod \"a5568fb6-d967-427d-9d42-26b57b1b05e9\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " Sep 06 06:20:04.844449 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.844301 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-data\") pod \"a5568fb6-d967-427d-9d42-26b57b1b05e9\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " Sep 06 06:20:04.844449 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.844323 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-workload-socket\") pod \"a5568fb6-d967-427d-9d42-26b57b1b05e9\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " Sep 06 06:20:04.844449 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.844351 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-workload-certs\") pod \"a5568fb6-d967-427d-9d42-26b57b1b05e9\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " Sep 06 06:20:04.844449 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.844400 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-token\") pod \"a5568fb6-d967-427d-9d42-26b57b1b05e9\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " Sep 06 06:20:04.844449 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.844442 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-credential-socket\") pod \"a5568fb6-d967-427d-9d42-26b57b1b05e9\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " Sep 06 06:20:04.844822 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.844500 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-envoy\") pod \"a5568fb6-d967-427d-9d42-26b57b1b05e9\" (UID: \"a5568fb6-d967-427d-9d42-26b57b1b05e9\") " Sep 06 06:20:04.844822 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.844626 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5568fb6-d967-427d-9d42-26b57b1b05e9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a5568fb6-d967-427d-9d42-26b57b1b05e9" (UID: "a5568fb6-d967-427d-9d42-26b57b1b05e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:20:04.844822 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.844726 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a5568fb6-d967-427d-9d42-26b57b1b05e9" (UID: "a5568fb6-d967-427d-9d42-26b57b1b05e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:20:04.844991 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.844817 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a5568fb6-d967-427d-9d42-26b57b1b05e9" (UID: "a5568fb6-d967-427d-9d42-26b57b1b05e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:20:04.845042 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.844991 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a5568fb6-d967-427d-9d42-26b57b1b05e9" (UID: "a5568fb6-d967-427d-9d42-26b57b1b05e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:20:04.845042 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.845006 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-data" (OuterVolumeSpecName: "istio-data") pod "a5568fb6-d967-427d-9d42-26b57b1b05e9" (UID: "a5568fb6-d967-427d-9d42-26b57b1b05e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:20:04.845321 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.845098 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5568fb6-d967-427d-9d42-26b57b1b05e9-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:20:04.845321 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.845118 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:20:04.845321 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.845131 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:20:04.846754 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.846723 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5568fb6-d967-427d-9d42-26b57b1b05e9-kube-api-access-whjzx" (OuterVolumeSpecName: "kube-api-access-whjzx") pod "a5568fb6-d967-427d-9d42-26b57b1b05e9" (UID: "a5568fb6-d967-427d-9d42-26b57b1b05e9"). InnerVolumeSpecName "kube-api-access-whjzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:20:04.846754 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.846734 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a5568fb6-d967-427d-9d42-26b57b1b05e9" (UID: "a5568fb6-d967-427d-9d42-26b57b1b05e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:20:04.846754 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.846738 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-token" (OuterVolumeSpecName: "istio-token") pod "a5568fb6-d967-427d-9d42-26b57b1b05e9" (UID: "a5568fb6-d967-427d-9d42-26b57b1b05e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:20:04.847200 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.847183 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a5568fb6-d967-427d-9d42-26b57b1b05e9" (UID: "a5568fb6-d967-427d-9d42-26b57b1b05e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:20:04.946322 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.946278 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whjzx\" (UniqueName: \"kubernetes.io/projected/a5568fb6-d967-427d-9d42-26b57b1b05e9-kube-api-access-whjzx\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:20:04.946322 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.946314 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:20:04.946322 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.946325 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:20:04.946322 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.946333 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:20:04.946598 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.946341 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:20:04.946598 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:04.946348 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5568fb6-d967-427d-9d42-26b57b1b05e9-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:20:05.115976 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:05.115877 2578 generic.go:356] "Generic (PLEG): container finished" podID="a5568fb6-d967-427d-9d42-26b57b1b05e9" containerID="3e87e3aea6115ebf339ee628594987c25a3eb7f8dbe24184874df111db8b3195" exitCode=0 Sep 06 06:20:05.115976 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:05.115958 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" Sep 06 06:20:05.116224 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:05.115970 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" event={"ID":"a5568fb6-d967-427d-9d42-26b57b1b05e9","Type":"ContainerDied","Data":"3e87e3aea6115ebf339ee628594987c25a3eb7f8dbe24184874df111db8b3195"} Sep 06 06:20:05.116224 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:05.116040 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg" event={"ID":"a5568fb6-d967-427d-9d42-26b57b1b05e9","Type":"ContainerDied","Data":"322e4fdb06df8b7896750b7f062e82bdfd0a2aa908de7f71f8b6c4ff934d72b5"} Sep 06 06:20:05.116224 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:05.116074 2578 scope.go:117] "RemoveContainer" containerID="3e87e3aea6115ebf339ee628594987c25a3eb7f8dbe24184874df111db8b3195" Sep 06 06:20:05.126948 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:05.126924 2578 scope.go:117] "RemoveContainer" containerID="66fc83fa2427833841781535e143302383478bb8b4d72a8a2490f4f3d69d4ea1" Sep 06 06:20:05.138028 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:05.138000 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg"] Sep 06 06:20:05.138715 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:05.138696 2578 scope.go:117] "RemoveContainer" containerID="3e87e3aea6115ebf339ee628594987c25a3eb7f8dbe24184874df111db8b3195" Sep 06 06:20:05.139091 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:20:05.139063 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e87e3aea6115ebf339ee628594987c25a3eb7f8dbe24184874df111db8b3195\": container with ID starting with 3e87e3aea6115ebf339ee628594987c25a3eb7f8dbe24184874df111db8b3195 not found: ID does not exist" containerID="3e87e3aea6115ebf339ee628594987c25a3eb7f8dbe24184874df111db8b3195" Sep 06 06:20:05.139194 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:05.139103 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e87e3aea6115ebf339ee628594987c25a3eb7f8dbe24184874df111db8b3195"} err="failed to get container status \"3e87e3aea6115ebf339ee628594987c25a3eb7f8dbe24184874df111db8b3195\": rpc error: code = NotFound desc = could not find container \"3e87e3aea6115ebf339ee628594987c25a3eb7f8dbe24184874df111db8b3195\": container with ID starting with 3e87e3aea6115ebf339ee628594987c25a3eb7f8dbe24184874df111db8b3195 not found: ID does not exist" Sep 06 06:20:05.139194 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:05.139124 2578 scope.go:117] "RemoveContainer" containerID="66fc83fa2427833841781535e143302383478bb8b4d72a8a2490f4f3d69d4ea1" Sep 06 06:20:05.139460 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:20:05.139441 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66fc83fa2427833841781535e143302383478bb8b4d72a8a2490f4f3d69d4ea1\": container with ID starting with 66fc83fa2427833841781535e143302383478bb8b4d72a8a2490f4f3d69d4ea1 not found: ID does not exist" containerID="66fc83fa2427833841781535e143302383478bb8b4d72a8a2490f4f3d69d4ea1" Sep 06 06:20:05.139513 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:05.139466 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66fc83fa2427833841781535e143302383478bb8b4d72a8a2490f4f3d69d4ea1"} err="failed to get container status \"66fc83fa2427833841781535e143302383478bb8b4d72a8a2490f4f3d69d4ea1\": rpc error: code = NotFound desc = could not find container \"66fc83fa2427833841781535e143302383478bb8b4d72a8a2490f4f3d69d4ea1\": container with ID starting with 66fc83fa2427833841781535e143302383478bb8b4d72a8a2490f4f3d69d4ea1 not found: ID does not exist" Sep 06 06:20:05.145289 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:05.145263 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-dd8c231a-openshift-default-d595fb58d-bhccg"] Sep 06 06:20:07.145770 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:07.145726 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5568fb6-d967-427d-9d42-26b57b1b05e9" path="/var/lib/kubelet/pods/a5568fb6-d967-427d-9d42-26b57b1b05e9/volumes" Sep 06 06:20:12.782247 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:12.782210 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:20:14.621299 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.621237 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" podUID="0425d06d-2049-4114-bdd2-c9328ad5aa97" containerName="main" containerID="cri-o://da43980fddf166b19a772ad4a95c95d54b11ccbacfd5c84c019a5266dfbea880" gracePeriod=45 Sep 06 06:20:14.876157 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.876092 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:20:14.944020 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.943983 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-tmp-dir\") pod \"0425d06d-2049-4114-bdd2-c9328ad5aa97\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " Sep 06 06:20:14.944215 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.944053 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-home\") pod \"0425d06d-2049-4114-bdd2-c9328ad5aa97\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " Sep 06 06:20:14.944215 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.944081 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd7zj\" (UniqueName: \"kubernetes.io/projected/0425d06d-2049-4114-bdd2-c9328ad5aa97-kube-api-access-cd7zj\") pod \"0425d06d-2049-4114-bdd2-c9328ad5aa97\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " Sep 06 06:20:14.944215 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.944127 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-dshm\") pod \"0425d06d-2049-4114-bdd2-c9328ad5aa97\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " Sep 06 06:20:14.944215 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.944149 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0425d06d-2049-4114-bdd2-c9328ad5aa97-tls-certs\") pod \"0425d06d-2049-4114-bdd2-c9328ad5aa97\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " Sep 06 06:20:14.944215 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.944192 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-model-cache\") pod \"0425d06d-2049-4114-bdd2-c9328ad5aa97\" (UID: \"0425d06d-2049-4114-bdd2-c9328ad5aa97\") " Sep 06 06:20:14.944483 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.944263 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0425d06d-2049-4114-bdd2-c9328ad5aa97" (UID: "0425d06d-2049-4114-bdd2-c9328ad5aa97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:20:14.944483 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.944325 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-home" (OuterVolumeSpecName: "home") pod "0425d06d-2049-4114-bdd2-c9328ad5aa97" (UID: "0425d06d-2049-4114-bdd2-c9328ad5aa97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:20:14.944564 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.944518 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-model-cache" (OuterVolumeSpecName: "model-cache") pod "0425d06d-2049-4114-bdd2-c9328ad5aa97" (UID: "0425d06d-2049-4114-bdd2-c9328ad5aa97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:20:14.944600 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.944584 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-home\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:20:14.944636 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.944602 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-tmp-dir\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:20:14.946518 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.946493 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-dshm" (OuterVolumeSpecName: "dshm") pod "0425d06d-2049-4114-bdd2-c9328ad5aa97" (UID: "0425d06d-2049-4114-bdd2-c9328ad5aa97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:20:14.946518 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.946501 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0425d06d-2049-4114-bdd2-c9328ad5aa97-kube-api-access-cd7zj" (OuterVolumeSpecName: "kube-api-access-cd7zj") pod "0425d06d-2049-4114-bdd2-c9328ad5aa97" (UID: "0425d06d-2049-4114-bdd2-c9328ad5aa97"). InnerVolumeSpecName "kube-api-access-cd7zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:20:14.946639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:14.946575 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0425d06d-2049-4114-bdd2-c9328ad5aa97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0425d06d-2049-4114-bdd2-c9328ad5aa97" (UID: "0425d06d-2049-4114-bdd2-c9328ad5aa97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:20:15.045823 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:15.045786 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-dshm\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:20:15.045823 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:15.045818 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0425d06d-2049-4114-bdd2-c9328ad5aa97-tls-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:20:15.045823 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:15.045829 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0425d06d-2049-4114-bdd2-c9328ad5aa97-model-cache\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:20:15.046118 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:15.045837 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd7zj\" (UniqueName: \"kubernetes.io/projected/0425d06d-2049-4114-bdd2-c9328ad5aa97-kube-api-access-cd7zj\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:20:15.174651 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:15.174615 2578 generic.go:356] "Generic (PLEG): container finished" podID="0425d06d-2049-4114-bdd2-c9328ad5aa97" containerID="da43980fddf166b19a772ad4a95c95d54b11ccbacfd5c84c019a5266dfbea880" exitCode=0 Sep 06 06:20:15.174844 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:15.174665 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" Sep 06 06:20:15.174844 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:15.174710 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" event={"ID":"0425d06d-2049-4114-bdd2-c9328ad5aa97","Type":"ContainerDied","Data":"da43980fddf166b19a772ad4a95c95d54b11ccbacfd5c84c019a5266dfbea880"} Sep 06 06:20:15.174844 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:15.174760 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp" event={"ID":"0425d06d-2049-4114-bdd2-c9328ad5aa97","Type":"ContainerDied","Data":"510f7d167b4e5ccd787e934fa2e2c3a4b1764d60ba4ef6f2bbbfda699bbb8d0f"} Sep 06 06:20:15.174844 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:15.174781 2578 scope.go:117] "RemoveContainer" containerID="da43980fddf166b19a772ad4a95c95d54b11ccbacfd5c84c019a5266dfbea880" Sep 06 06:20:15.183712 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:15.183690 2578 scope.go:117] "RemoveContainer" containerID="da43980fddf166b19a772ad4a95c95d54b11ccbacfd5c84c019a5266dfbea880" Sep 06 06:20:15.184041 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:20:15.184009 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da43980fddf166b19a772ad4a95c95d54b11ccbacfd5c84c019a5266dfbea880\": container with ID starting with da43980fddf166b19a772ad4a95c95d54b11ccbacfd5c84c019a5266dfbea880 not found: ID does not exist" containerID="da43980fddf166b19a772ad4a95c95d54b11ccbacfd5c84c019a5266dfbea880" Sep 06 06:20:15.184145 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:15.184046 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da43980fddf166b19a772ad4a95c95d54b11ccbacfd5c84c019a5266dfbea880"} err="failed to get container status \"da43980fddf166b19a772ad4a95c95d54b11ccbacfd5c84c019a5266dfbea880\": rpc error: code = NotFound desc = could not find container \"da43980fddf166b19a772ad4a95c95d54b11ccbacfd5c84c019a5266dfbea880\": container with ID starting with da43980fddf166b19a772ad4a95c95d54b11ccbacfd5c84c019a5266dfbea880 not found: ID does not exist" Sep 06 06:20:15.190667 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:15.190642 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp"] Sep 06 06:20:15.194493 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:15.194470 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dc97f9/auth-test-model-dc97f9-kserve-7bf86bbf75-6g5kp"] Sep 06 06:20:17.146438 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:17.146399 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0425d06d-2049-4114-bdd2-c9328ad5aa97" path="/var/lib/kubelet/pods/0425d06d-2049-4114-bdd2-c9328ad5aa97/volumes" Sep 06 06:20:40.309109 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.309078 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452"] Sep 06 06:20:40.309577 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.309471 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec" containerName="mt-curl-7728-e83b72" Sep 06 06:20:40.309577 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.309482 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec" containerName="mt-curl-7728-e83b72" Sep 06 06:20:40.309577 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.309493 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5568fb6-d967-427d-9d42-26b57b1b05e9" containerName="istio-proxy" Sep 06 06:20:40.309577 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.309498 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5568fb6-d967-427d-9d42-26b57b1b05e9" containerName="istio-proxy" Sep 06 06:20:40.309577 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.309509 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5568fb6-d967-427d-9d42-26b57b1b05e9" containerName="istio-proxy" Sep 06 06:20:40.309577 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.309514 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5568fb6-d967-427d-9d42-26b57b1b05e9" containerName="istio-proxy" Sep 06 06:20:40.309577 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.309523 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0425d06d-2049-4114-bdd2-c9328ad5aa97" containerName="main" Sep 06 06:20:40.309577 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.309528 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0425d06d-2049-4114-bdd2-c9328ad5aa97" containerName="main" Sep 06 06:20:40.309828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.309605 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5568fb6-d967-427d-9d42-26b57b1b05e9" containerName="istio-proxy" Sep 06 06:20:40.309828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.309620 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5568fb6-d967-427d-9d42-26b57b1b05e9" containerName="istio-proxy" Sep 06 06:20:40.309828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.309631 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0425d06d-2049-4114-bdd2-c9328ad5aa97" containerName="main" Sep 06 06:20:40.309828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.309640 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94ba603-9357-4dcf-a1c7-7a6eb0c8cbec" containerName="mt-curl-7728-e83b72" Sep 06 06:20:40.314076 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.314058 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.316376 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.316354 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-429831\"/\"sub-test-model-429831-kserve-self-signed-certs\"" Sep 06 06:20:40.317035 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.317015 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-429831\"/\"openshift-service-ca.crt\"" Sep 06 06:20:40.317140 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.317065 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-429831\"/\"default-dockercfg-2vjht\"" Sep 06 06:20:40.317140 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.317089 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-429831\"/\"kube-root-ca.crt\"" Sep 06 06:20:40.323515 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.323482 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452"] Sep 06 06:20:40.498239 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.498196 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxbtf\" (UniqueName: \"kubernetes.io/projected/591c6655-156a-44c5-a59c-a20abe555780-kube-api-access-dxbtf\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.498460 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.498258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591c6655-156a-44c5-a59c-a20abe555780-tls-certs\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.498460 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.498364 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-home\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.498460 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.498436 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-model-cache\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.498602 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.498471 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-tmp-dir\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.498602 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.498537 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-dshm\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.599921 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.599799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-home\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.599921 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.599859 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-model-cache\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.599921 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.599879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-tmp-dir\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.600224 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.599950 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-dshm\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.600224 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.600014 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxbtf\" (UniqueName: \"kubernetes.io/projected/591c6655-156a-44c5-a59c-a20abe555780-kube-api-access-dxbtf\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.600224 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.600045 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591c6655-156a-44c5-a59c-a20abe555780-tls-certs\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.600382 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.600270 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-home\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.600382 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.600336 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-tmp-dir\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.600382 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.600333 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-model-cache\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.602538 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.602520 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-dshm\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.602721 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.602704 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591c6655-156a-44c5-a59c-a20abe555780-tls-certs\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.607662 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.607644 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxbtf\" (UniqueName: \"kubernetes.io/projected/591c6655-156a-44c5-a59c-a20abe555780-kube-api-access-dxbtf\") pod \"sub-test-model-429831-kserve-69885f5c69-52452\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.625460 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.625434 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:40.900867 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:40.900842 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452"] Sep 06 06:20:40.903075 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:20:40.903035 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod591c6655_156a_44c5_a59c_a20abe555780.slice/crio-010ceeaa7fdcdf26ff0e713f80ce174fcaa04820d714cbc7703390993d63d8ed WatchSource:0}: Error finding container 010ceeaa7fdcdf26ff0e713f80ce174fcaa04820d714cbc7703390993d63d8ed: Status 404 returned error can't find the container with id 010ceeaa7fdcdf26ff0e713f80ce174fcaa04820d714cbc7703390993d63d8ed Sep 06 06:20:41.340444 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:41.340407 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" event={"ID":"591c6655-156a-44c5-a59c-a20abe555780","Type":"ContainerStarted","Data":"374dc1441d074c7090fddc157dedca944500d7020fe51407eb61f8aa6b17fd21"} Sep 06 06:20:41.340444 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:41.340449 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" event={"ID":"591c6655-156a-44c5-a59c-a20abe555780","Type":"ContainerStarted","Data":"010ceeaa7fdcdf26ff0e713f80ce174fcaa04820d714cbc7703390993d63d8ed"} Sep 06 06:20:41.356232 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:41.356174 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" podStartSLOduration=1.356153646 podStartE2EDuration="1.356153646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:20:41.355116399 +0000 UTC m=+1606.753446875" watchObservedRunningTime="2026-09-06 06:20:41.356153646 +0000 UTC m=+1606.754484114" Sep 06 06:20:44.276470 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:44.276434 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:20:50.625987 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:50.625870 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:50.625987 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:50.625964 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:50.638317 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:50.638285 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:51.406317 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:51.406290 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:20:53.089201 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.089172 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm"] Sep 06 06:20:53.093002 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.092978 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.095190 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.095168 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ed85e8-openshift-default-dockercfg-7swgx\"" Sep 06 06:20:53.104253 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.104225 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm"] Sep 06 06:20:53.127641 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.127600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-credential-socket\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.127641 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.127638 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-data\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.127886 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.127662 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-podinfo\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.127886 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.127761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwlb5\" (UniqueName: \"kubernetes.io/projected/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-kube-api-access-wwlb5\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.127886 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.127792 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-workload-certs\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.127886 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.127860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-envoy\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.127886 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.127880 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-token\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.128198 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.127942 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-workload-socket\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.128198 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.127992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istiod-ca-cert\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.228973 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.228930 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-envoy\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.228973 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.228978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-token\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.229229 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.229023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-workload-socket\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.229229 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.229053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istiod-ca-cert\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.229229 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.229095 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-credential-socket\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.229229 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.229127 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-data\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.229229 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.229159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-podinfo\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.229229 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.229227 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlb5\" (UniqueName: \"kubernetes.io/projected/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-kube-api-access-wwlb5\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.229473 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.229261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-workload-certs\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.229523 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.229499 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-workload-socket\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.229771 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.229746 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istiod-ca-cert\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.229945 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.229815 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-credential-socket\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.229945 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.229825 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-data\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.230127 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.230107 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-workload-certs\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.231573 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.231543 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-envoy\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.231728 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.231705 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-podinfo\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.236394 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.236373 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-token\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.236494 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.236421 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwlb5\" (UniqueName: \"kubernetes.io/projected/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-kube-api-access-wwlb5\") pod \"e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.405069 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.404976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:53.680229 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.680199 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm"] Sep 06 06:20:53.682092 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:20:53.682069 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04fe53c_e5b0_4d5a_a064_f9ed623f6ed1.slice/crio-e0da0b4bd6918d1338520104a35197a2465ef7d9a2eee6ce2de0b36f7fe8994f WatchSource:0}: Error finding container e0da0b4bd6918d1338520104a35197a2465ef7d9a2eee6ce2de0b36f7fe8994f: Status 404 returned error can't find the container with id e0da0b4bd6918d1338520104a35197a2465ef7d9a2eee6ce2de0b36f7fe8994f Sep 06 06:20:53.685388 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.685352 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:20:53.685631 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.685590 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:20:53.685790 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:53.685772 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:20:54.416731 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:54.416686 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" event={"ID":"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1","Type":"ContainerStarted","Data":"bfa28841c5d52be1b8f3c87ed1617d13dfe28a574e9e0252f4f7692eda6b796a"} Sep 06 06:20:54.417116 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:54.416753 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" event={"ID":"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1","Type":"ContainerStarted","Data":"e0da0b4bd6918d1338520104a35197a2465ef7d9a2eee6ce2de0b36f7fe8994f"} Sep 06 06:20:54.435272 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:54.435221 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" podStartSLOduration=1.435206762 podStartE2EDuration="1.435206762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:20:54.433518412 +0000 UTC m=+1619.831848935" watchObservedRunningTime="2026-09-06 06:20:54.435206762 +0000 UTC m=+1619.833537222" Sep 06 06:20:55.405364 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:55.405326 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:55.407854 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:55.407823 2578 scope.go:117] "RemoveContainer" containerID="d249e990103876aac41ca6ad15f76df626792a21de0264875c27ab7e79c8fb1a" Sep 06 06:20:55.411192 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:55.411168 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:55.423299 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:55.423279 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:20:55.424091 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:20:55.424074 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:21:05.190540 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:05.190506 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:21:15.546158 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:15.546125 2578 generic.go:356] "Generic (PLEG): container finished" podID="c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" containerID="bfa28841c5d52be1b8f3c87ed1617d13dfe28a574e9e0252f4f7692eda6b796a" exitCode=0 Sep 06 06:21:15.546576 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:15.546198 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" event={"ID":"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1","Type":"ContainerDied","Data":"bfa28841c5d52be1b8f3c87ed1617d13dfe28a574e9e0252f4f7692eda6b796a"} Sep 06 06:21:15.546623 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:15.546611 2578 scope.go:117] "RemoveContainer" containerID="bfa28841c5d52be1b8f3c87ed1617d13dfe28a574e9e0252f4f7692eda6b796a" Sep 06 06:21:15.547789 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:15.547760 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:21:15.547861 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:15.547840 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:21:15.547898 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:15.547872 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:21:16.554384 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:16.554350 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" event={"ID":"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1","Type":"ContainerStarted","Data":"d10ecf1bc9dcba4db7f24c3a42ee0491688b4a4def37fa1770fb56b53c735ea4"} Sep 06 06:21:17.406076 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:17.406041 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:21:17.406424 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:17.406394 2578 patch_prober.go:28] interesting pod/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 06 06:21:17.406544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:17.406458 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" podUID="c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 06 06:21:17.602251 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:17.602217 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:21:18.405343 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:18.405306 2578 patch_prober.go:28] interesting pod/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 06 06:21:18.405499 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:18.405362 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" podUID="c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 06 06:21:19.406043 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:19.406001 2578 patch_prober.go:28] interesting pod/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 06 06:21:19.406441 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:19.406078 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" podUID="c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 06 06:21:20.405517 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:20.405481 2578 patch_prober.go:28] interesting pod/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 06 06:21:20.405693 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:20.405533 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" podUID="c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 06 06:21:21.406784 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:21.406751 2578 patch_prober.go:28] interesting pod/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 06 06:21:21.407235 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:21.406804 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" podUID="c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 06 06:21:22.105248 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.105199 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d"] Sep 06 06:21:22.109079 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.109057 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.111240 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.111215 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1c55e7-openshift-default-dockercfg-vw6q2\"" Sep 06 06:21:22.121736 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.121709 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d"] Sep 06 06:21:22.130119 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.130094 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-envoy\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.130249 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.130142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-data\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.130249 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.130214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-token\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.130398 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.130315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istiod-ca-cert\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.130398 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.130364 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-credential-socket\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.130725 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.130399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-workload-certs\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.130725 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.130469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-podinfo\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.130725 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.130508 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-workload-socket\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.130725 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.130533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkjtw\" (UniqueName: \"kubernetes.io/projected/5e2fe0a4-ec00-4a32-bdfa-da789382de17-kube-api-access-pkjtw\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.232088 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.232049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istiod-ca-cert\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.232372 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.232352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-credential-socket\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.232815 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.232797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-workload-certs\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.233030 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.233012 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-podinfo\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.233289 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.233262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-workload-socket\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.233385 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.233307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkjtw\" (UniqueName: \"kubernetes.io/projected/5e2fe0a4-ec00-4a32-bdfa-da789382de17-kube-api-access-pkjtw\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.233385 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.233349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-envoy\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.233480 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.233417 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-data\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.233480 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.232874 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istiod-ca-cert\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.233480 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.233452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-token\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.233480 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.232734 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-credential-socket\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.233689 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.233131 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-workload-certs\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.233823 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.233757 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-workload-socket\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.234107 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.234085 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-data\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.236640 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.236607 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-podinfo\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.237049 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.237030 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-envoy\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.242728 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.242702 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-token\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.243055 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.243030 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkjtw\" (UniqueName: \"kubernetes.io/projected/5e2fe0a4-ec00-4a32-bdfa-da789382de17-kube-api-access-pkjtw\") pod \"e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.423180 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.423094 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:22.554925 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.554875 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:21:22.555357 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.555334 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:21:22.556393 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.556373 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:21:22.702364 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.702319 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d"] Sep 06 06:21:22.703995 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:21:22.703967 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2fe0a4_ec00_4a32_bdfa_da789382de17.slice/crio-22195abdfef9b5651376cee5979831fe9b4ffa329d54cafe30498988b8a2f200 WatchSource:0}: Error finding container 22195abdfef9b5651376cee5979831fe9b4ffa329d54cafe30498988b8a2f200: Status 404 returned error can't find the container with id 22195abdfef9b5651376cee5979831fe9b4ffa329d54cafe30498988b8a2f200 Sep 06 06:21:22.706730 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.706698 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:21:22.706840 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.706774 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:21:22.706840 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:22.706820 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:21:23.606662 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:23.606626 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" event={"ID":"5e2fe0a4-ec00-4a32-bdfa-da789382de17","Type":"ContainerStarted","Data":"89f8a454041d1aae5e1d9afb0e253f374734d4fc05e1dd23f4b3528b807cfb52"} Sep 06 06:21:23.607137 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:23.606689 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" event={"ID":"5e2fe0a4-ec00-4a32-bdfa-da789382de17","Type":"ContainerStarted","Data":"22195abdfef9b5651376cee5979831fe9b4ffa329d54cafe30498988b8a2f200"} Sep 06 06:21:23.627346 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:23.627303 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" podStartSLOduration=1.62728768 podStartE2EDuration="1.62728768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:21:23.624639421 +0000 UTC m=+1649.022969886" watchObservedRunningTime="2026-09-06 06:21:23.62728768 +0000 UTC m=+1649.025618146" Sep 06 06:21:24.423871 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:24.423834 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:24.428933 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:24.428882 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:24.612339 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:24.612308 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:24.613303 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:24.613284 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:26.692632 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.692597 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr"] Sep 06 06:21:26.696986 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.696967 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:26.699253 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.699229 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1c55e7-serving-cert\"" Sep 06 06:21:26.699386 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.699240 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 06 06:21:26.699919 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.699883 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5vd8t\"" Sep 06 06:21:26.700020 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.699929 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:21:26.700020 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.699888 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 06 06:21:26.704212 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.704188 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr"] Sep 06 06:21:26.781947 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.781865 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:26.782136 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.781959 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5chrk\" (UniqueName: \"kubernetes.io/projected/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-kube-api-access-5chrk\") pod \"maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:26.782136 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.782004 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-metrics-certs\") pod \"maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:26.782136 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.782036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-maas-api-tls\") pod \"maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:26.883547 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.883510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:26.883547 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.883546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5chrk\" (UniqueName: \"kubernetes.io/projected/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-kube-api-access-5chrk\") pod \"maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:26.883756 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.883575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-metrics-certs\") pod \"maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:26.883756 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.883618 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-maas-api-tls\") pod \"maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:26.884190 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.884167 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:26.886271 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.886248 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-metrics-certs\") pod \"maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:26.886387 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.886295 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-maas-api-tls\") pod \"maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:26.892266 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:26.892230 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5chrk\" (UniqueName: \"kubernetes.io/projected/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-kube-api-access-5chrk\") pod \"maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:27.011346 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:27.011257 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:27.286847 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:27.286816 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr"] Sep 06 06:21:27.289194 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:21:27.289174 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67934d5d_048f_4a3c_b02f_6f65d04c2fbb.slice/crio-541be4285de0ff519a673552c3376873b6230ae8e319cadb449ece12404aff3c WatchSource:0}: Error finding container 541be4285de0ff519a673552c3376873b6230ae8e319cadb449ece12404aff3c: Status 404 returned error can't find the container with id 541be4285de0ff519a673552c3376873b6230ae8e319cadb449ece12404aff3c Sep 06 06:21:27.631434 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:27.631349 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" event={"ID":"67934d5d-048f-4a3c-b02f-6f65d04c2fbb","Type":"ContainerStarted","Data":"541be4285de0ff519a673552c3376873b6230ae8e319cadb449ece12404aff3c"} Sep 06 06:21:29.646544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:29.646515 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" event={"ID":"67934d5d-048f-4a3c-b02f-6f65d04c2fbb","Type":"ContainerStarted","Data":"7fa0c45c1d27a55fd725a8c96216e26912b9af96bf22f33ef52931ab3f0d43e0"} Sep 06 06:21:29.646997 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:29.646670 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:29.663256 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:29.663212 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" podStartSLOduration=2.194355306 podStartE2EDuration="3.663198142s" totalImagesPullingTime="1.468842836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:21:29.661147739 +0000 UTC m=+1655.059478216" watchObservedRunningTime="2026-09-06 06:21:29.663198142 +0000 UTC m=+1655.061528609" Sep 06 06:21:35.079630 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:35.079594 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:21:35.665182 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:35.665146 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:21:44.745489 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:44.744978 2578 generic.go:356] "Generic (PLEG): container finished" podID="5e2fe0a4-ec00-4a32-bdfa-da789382de17" containerID="89f8a454041d1aae5e1d9afb0e253f374734d4fc05e1dd23f4b3528b807cfb52" exitCode=0 Sep 06 06:21:44.745489 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:44.745080 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" event={"ID":"5e2fe0a4-ec00-4a32-bdfa-da789382de17","Type":"ContainerDied","Data":"89f8a454041d1aae5e1d9afb0e253f374734d4fc05e1dd23f4b3528b807cfb52"} Sep 06 06:21:44.746103 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:44.745707 2578 scope.go:117] "RemoveContainer" containerID="89f8a454041d1aae5e1d9afb0e253f374734d4fc05e1dd23f4b3528b807cfb52" Sep 06 06:21:44.747265 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:44.747235 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:21:44.747372 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:44.747318 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:21:44.747372 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:44.747365 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:21:45.754074 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:45.754033 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" event={"ID":"5e2fe0a4-ec00-4a32-bdfa-da789382de17","Type":"ContainerStarted","Data":"54f222113e8c715cbd59d0016a561a94878800bbd134fd644f72c2340783652a"} Sep 06 06:21:46.423666 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:46.423622 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:46.423869 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:46.423851 2578 patch_prober.go:28] interesting pod/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 06 06:21:46.423969 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:46.423947 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" podUID="5e2fe0a4-ec00-4a32-bdfa-da789382de17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 06 06:21:47.424238 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:47.424200 2578 patch_prober.go:28] interesting pod/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 06 06:21:47.424620 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:47.424265 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" podUID="5e2fe0a4-ec00-4a32-bdfa-da789382de17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 06 06:21:48.424400 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:48.424359 2578 patch_prober.go:28] interesting pod/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 06 06:21:48.424801 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:48.424441 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" podUID="5e2fe0a4-ec00-4a32-bdfa-da789382de17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 06 06:21:49.424184 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:49.424149 2578 patch_prober.go:28] interesting pod/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 06 06:21:49.424366 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:49.424205 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" podUID="5e2fe0a4-ec00-4a32-bdfa-da789382de17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 06 06:21:50.424134 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:50.424085 2578 patch_prober.go:28] interesting pod/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Sep 06 06:21:50.424619 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:50.424154 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" podUID="5e2fe0a4-ec00-4a32-bdfa-da789382de17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Sep 06 06:21:51.428502 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:51.428473 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:51.428937 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:51.428917 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:51.429739 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:51.429719 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:21:55.440676 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:55.440643 2578 scope.go:117] "RemoveContainer" containerID="6e559a269130d273ca98571862468c908344da0056cc031284e1f62b093e660a" Sep 06 06:21:55.449532 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:55.449513 2578 scope.go:117] "RemoveContainer" containerID="b5ea4ec3413bd3acf49ef3f3c98365ff022c5f3224836a26f01250a19117b01b" Sep 06 06:21:55.457728 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:55.457706 2578 scope.go:117] "RemoveContainer" containerID="20f982f7f42d7ce9fd699eacfccd58cd9549f60c3d7e0a1db2585d205213f346" Sep 06 06:21:55.468217 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:21:55.468192 2578 scope.go:117] "RemoveContainer" containerID="490622ecc70a87ad5a124b2084bbecb47f4fc7b30d661c8a859efc0acc91c94a" Sep 06 06:22:08.376706 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:08.376672 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:22:12.163748 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:12.163706 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7728-3267f2"] Sep 06 06:22:12.168832 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:12.168807 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7728-3267f2" Sep 06 06:22:12.171306 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:12.171279 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-j266t\"" Sep 06 06:22:12.176248 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:12.176221 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7728-3267f2"] Sep 06 06:22:12.259377 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:12.259338 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f444q\" (UniqueName: \"kubernetes.io/projected/cf29b759-1b01-4cd7-942b-ecea5e69d8a6-kube-api-access-f444q\") pod \"mt-curl-7728-3267f2\" (UID: \"cf29b759-1b01-4cd7-942b-ecea5e69d8a6\") " pod="opendatahub/mt-curl-7728-3267f2" Sep 06 06:22:12.360156 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:12.360116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f444q\" (UniqueName: \"kubernetes.io/projected/cf29b759-1b01-4cd7-942b-ecea5e69d8a6-kube-api-access-f444q\") pod \"mt-curl-7728-3267f2\" (UID: \"cf29b759-1b01-4cd7-942b-ecea5e69d8a6\") " pod="opendatahub/mt-curl-7728-3267f2" Sep 06 06:22:12.367921 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:12.367881 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f444q\" (UniqueName: \"kubernetes.io/projected/cf29b759-1b01-4cd7-942b-ecea5e69d8a6-kube-api-access-f444q\") pod \"mt-curl-7728-3267f2\" (UID: \"cf29b759-1b01-4cd7-942b-ecea5e69d8a6\") " pod="opendatahub/mt-curl-7728-3267f2" Sep 06 06:22:12.486877 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:12.486791 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7728-3267f2" Sep 06 06:22:12.755158 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:12.755134 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7728-3267f2"] Sep 06 06:22:12.757546 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:22:12.757507 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf29b759_1b01_4cd7_942b_ecea5e69d8a6.slice/crio-23bb5542a2d868a32dcb6eb2086709d2e5810e0b79e783e4d3737552bd718f4a WatchSource:0}: Error finding container 23bb5542a2d868a32dcb6eb2086709d2e5810e0b79e783e4d3737552bd718f4a: Status 404 returned error can't find the container with id 23bb5542a2d868a32dcb6eb2086709d2e5810e0b79e783e4d3737552bd718f4a Sep 06 06:22:12.759352 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:12.759328 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 06:22:12.914054 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:12.914021 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7728-3267f2" event={"ID":"cf29b759-1b01-4cd7-942b-ecea5e69d8a6","Type":"ContainerStarted","Data":"23bb5542a2d868a32dcb6eb2086709d2e5810e0b79e783e4d3737552bd718f4a"} Sep 06 06:22:13.920648 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:13.920612 2578 generic.go:356] "Generic (PLEG): container finished" podID="cf29b759-1b01-4cd7-942b-ecea5e69d8a6" containerID="d9697b8e4fd6f8a376554882bc15326c086bffc790638d1eb402ae3d98df259f" exitCode=0 Sep 06 06:22:13.921046 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:13.920699 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7728-3267f2" event={"ID":"cf29b759-1b01-4cd7-942b-ecea5e69d8a6","Type":"ContainerDied","Data":"d9697b8e4fd6f8a376554882bc15326c086bffc790638d1eb402ae3d98df259f"} Sep 06 06:22:15.061067 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.061043 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7728-3267f2" Sep 06 06:22:15.091526 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.091502 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7728-3267f2_cf29b759-1b01-4cd7-942b-ecea5e69d8a6/mt-curl-7728-3267f2/0.log" Sep 06 06:22:15.111316 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.111286 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7728-3267f2"] Sep 06 06:22:15.114364 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.114335 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7728-3267f2"] Sep 06 06:22:15.191671 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.191590 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f444q\" (UniqueName: \"kubernetes.io/projected/cf29b759-1b01-4cd7-942b-ecea5e69d8a6-kube-api-access-f444q\") pod \"cf29b759-1b01-4cd7-942b-ecea5e69d8a6\" (UID: \"cf29b759-1b01-4cd7-942b-ecea5e69d8a6\") " Sep 06 06:22:15.193822 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.193794 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf29b759-1b01-4cd7-942b-ecea5e69d8a6-kube-api-access-f444q" (OuterVolumeSpecName: "kube-api-access-f444q") pod "cf29b759-1b01-4cd7-942b-ecea5e69d8a6" (UID: "cf29b759-1b01-4cd7-942b-ecea5e69d8a6"). InnerVolumeSpecName "kube-api-access-f444q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:22:15.229037 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.229005 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7728-ef04a0"] Sep 06 06:22:15.229424 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.229410 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf29b759-1b01-4cd7-942b-ecea5e69d8a6" containerName="mt-curl-7728-3267f2" Sep 06 06:22:15.229473 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.229425 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf29b759-1b01-4cd7-942b-ecea5e69d8a6" containerName="mt-curl-7728-3267f2" Sep 06 06:22:15.229508 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.229491 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf29b759-1b01-4cd7-942b-ecea5e69d8a6" containerName="mt-curl-7728-3267f2" Sep 06 06:22:15.232495 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.232479 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7728-ef04a0" Sep 06 06:22:15.237730 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.237700 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7728-ef04a0"] Sep 06 06:22:15.292820 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.292782 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f444q\" (UniqueName: \"kubernetes.io/projected/cf29b759-1b01-4cd7-942b-ecea5e69d8a6-kube-api-access-f444q\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:15.393692 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.393655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ghn\" (UniqueName: \"kubernetes.io/projected/e893f445-e42e-450c-8934-01f038d21b19-kube-api-access-v8ghn\") pod \"mt-curl-7728-ef04a0\" (UID: \"e893f445-e42e-450c-8934-01f038d21b19\") " pod="opendatahub/mt-curl-7728-ef04a0" Sep 06 06:22:15.494730 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.494634 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ghn\" (UniqueName: \"kubernetes.io/projected/e893f445-e42e-450c-8934-01f038d21b19-kube-api-access-v8ghn\") pod \"mt-curl-7728-ef04a0\" (UID: \"e893f445-e42e-450c-8934-01f038d21b19\") " pod="opendatahub/mt-curl-7728-ef04a0" Sep 06 06:22:15.502386 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.502360 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ghn\" (UniqueName: \"kubernetes.io/projected/e893f445-e42e-450c-8934-01f038d21b19-kube-api-access-v8ghn\") pod \"mt-curl-7728-ef04a0\" (UID: \"e893f445-e42e-450c-8934-01f038d21b19\") " pod="opendatahub/mt-curl-7728-ef04a0" Sep 06 06:22:15.542614 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.542576 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7728-ef04a0" Sep 06 06:22:15.839925 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.839385 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7728-ef04a0"] Sep 06 06:22:15.849637 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:22:15.849598 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode893f445_e42e_450c_8934_01f038d21b19.slice/crio-da3959dcc28dee2b86cb0478ff785e971925822def3365f9c10230f8e0c7ba33 WatchSource:0}: Error finding container da3959dcc28dee2b86cb0478ff785e971925822def3365f9c10230f8e0c7ba33: Status 404 returned error can't find the container with id da3959dcc28dee2b86cb0478ff785e971925822def3365f9c10230f8e0c7ba33 Sep 06 06:22:15.935646 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.935608 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7728-3267f2" Sep 06 06:22:15.935646 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.935643 2578 scope.go:117] "RemoveContainer" containerID="d9697b8e4fd6f8a376554882bc15326c086bffc790638d1eb402ae3d98df259f" Sep 06 06:22:15.937454 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:15.937426 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7728-ef04a0" event={"ID":"e893f445-e42e-450c-8934-01f038d21b19","Type":"ContainerStarted","Data":"da3959dcc28dee2b86cb0478ff785e971925822def3365f9c10230f8e0c7ba33"} Sep 06 06:22:16.034615 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:16.034576 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm"] Sep 06 06:22:16.034859 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:16.034822 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" podUID="c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" containerName="istio-proxy" containerID="cri-o://d10ecf1bc9dcba4db7f24c3a42ee0491688b4a4def37fa1770fb56b53c735ea4" gracePeriod=30 Sep 06 06:22:16.944267 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:16.944226 2578 generic.go:356] "Generic (PLEG): container finished" podID="e893f445-e42e-450c-8934-01f038d21b19" containerID="cfa26524bf4aec1860efaa83f58897ae2ba8c93f450d45bc3f0cbc96825b5c40" exitCode=0 Sep 06 06:22:16.944701 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:16.944298 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7728-ef04a0" event={"ID":"e893f445-e42e-450c-8934-01f038d21b19","Type":"ContainerDied","Data":"cfa26524bf4aec1860efaa83f58897ae2ba8c93f450d45bc3f0cbc96825b5c40"} Sep 06 06:22:17.145618 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:17.145575 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf29b759-1b01-4cd7-942b-ecea5e69d8a6" path="/var/lib/kubelet/pods/cf29b759-1b01-4cd7-942b-ecea5e69d8a6/volumes" Sep 06 06:22:18.091510 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:18.091487 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7728-ef04a0" Sep 06 06:22:18.109583 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:18.109554 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7728-ef04a0_e893f445-e42e-450c-8934-01f038d21b19/mt-curl-7728-ef04a0/0.log" Sep 06 06:22:18.134212 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:18.134170 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7728-ef04a0"] Sep 06 06:22:18.136128 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:18.136108 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7728-ef04a0"] Sep 06 06:22:18.221616 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:18.221515 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8ghn\" (UniqueName: \"kubernetes.io/projected/e893f445-e42e-450c-8934-01f038d21b19-kube-api-access-v8ghn\") pod \"e893f445-e42e-450c-8934-01f038d21b19\" (UID: \"e893f445-e42e-450c-8934-01f038d21b19\") " Sep 06 06:22:18.223733 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:18.223704 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e893f445-e42e-450c-8934-01f038d21b19-kube-api-access-v8ghn" (OuterVolumeSpecName: "kube-api-access-v8ghn") pod "e893f445-e42e-450c-8934-01f038d21b19" (UID: "e893f445-e42e-450c-8934-01f038d21b19"). InnerVolumeSpecName "kube-api-access-v8ghn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:22:18.322780 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:18.322737 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8ghn\" (UniqueName: \"kubernetes.io/projected/e893f445-e42e-450c-8934-01f038d21b19-kube-api-access-v8ghn\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:18.958751 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:18.958725 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7728-ef04a0" Sep 06 06:22:18.958977 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:18.958755 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da3959dcc28dee2b86cb0478ff785e971925822def3365f9c10230f8e0c7ba33" Sep 06 06:22:18.968137 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:22:18.968107 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7728-ef04a0\" is forbidden: User \"system:node:ip-10-0-135-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-32.ec2.internal' and this object" podUID="e893f445-e42e-450c-8934-01f038d21b19" pod="opendatahub/mt-curl-7728-ef04a0" Sep 06 06:22:19.146086 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:19.146051 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e893f445-e42e-450c-8934-01f038d21b19" path="/var/lib/kubelet/pods/e893f445-e42e-450c-8934-01f038d21b19/volumes" Sep 06 06:22:20.096934 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:20.096886 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452"] Sep 06 06:22:21.306959 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.306873 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:22:21.457179 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.457151 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-token\") pod \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " Sep 06 06:22:21.457347 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.457194 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-workload-socket\") pod \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " Sep 06 06:22:21.457347 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.457231 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwlb5\" (UniqueName: \"kubernetes.io/projected/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-kube-api-access-wwlb5\") pod \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " Sep 06 06:22:21.457347 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.457276 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-credential-socket\") pod \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " Sep 06 06:22:21.457465 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.457445 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-data\") pod \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " Sep 06 06:22:21.457517 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.457498 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-envoy\") pod \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " Sep 06 06:22:21.457568 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.457524 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" (UID: "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:21.457568 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.457551 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" (UID: "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:21.457683 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.457539 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-podinfo\") pod \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " Sep 06 06:22:21.457683 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.457613 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-workload-certs\") pod \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " Sep 06 06:22:21.457683 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.457639 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istiod-ca-cert\") pod \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\" (UID: \"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1\") " Sep 06 06:22:21.457865 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.457842 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" (UID: "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:21.458216 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.458189 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-data" (OuterVolumeSpecName: "istio-data") pod "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" (UID: "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:21.458515 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.458319 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" (UID: "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:22:21.458638 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.458296 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:21.458766 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.458652 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:21.458766 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.458671 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:21.459604 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.459581 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-token" (OuterVolumeSpecName: "istio-token") pod "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" (UID: "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:22:21.459725 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.459704 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" (UID: "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:22:21.459799 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.459705 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-kube-api-access-wwlb5" (OuterVolumeSpecName: "kube-api-access-wwlb5") pod "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" (UID: "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1"). InnerVolumeSpecName "kube-api-access-wwlb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:22:21.459889 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.459874 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" (UID: "c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:21.559667 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.559580 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:21.559667 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.559612 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:21.559667 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.559620 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:21.559667 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.559628 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:21.559667 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.559636 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwlb5\" (UniqueName: \"kubernetes.io/projected/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-kube-api-access-wwlb5\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:21.559667 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.559643 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:21.977763 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.977675 2578 generic.go:356] "Generic (PLEG): container finished" podID="c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" containerID="d10ecf1bc9dcba4db7f24c3a42ee0491688b4a4def37fa1770fb56b53c735ea4" exitCode=0 Sep 06 06:22:21.977763 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.977729 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" Sep 06 06:22:21.978045 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.977757 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" event={"ID":"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1","Type":"ContainerDied","Data":"d10ecf1bc9dcba4db7f24c3a42ee0491688b4a4def37fa1770fb56b53c735ea4"} Sep 06 06:22:21.978045 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.977810 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm" event={"ID":"c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1","Type":"ContainerDied","Data":"e0da0b4bd6918d1338520104a35197a2465ef7d9a2eee6ce2de0b36f7fe8994f"} Sep 06 06:22:21.978045 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.977831 2578 scope.go:117] "RemoveContainer" containerID="d10ecf1bc9dcba4db7f24c3a42ee0491688b4a4def37fa1770fb56b53c735ea4" Sep 06 06:22:21.987996 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.987980 2578 scope.go:117] "RemoveContainer" containerID="bfa28841c5d52be1b8f3c87ed1617d13dfe28a574e9e0252f4f7692eda6b796a" Sep 06 06:22:21.999481 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.999465 2578 scope.go:117] "RemoveContainer" containerID="d10ecf1bc9dcba4db7f24c3a42ee0491688b4a4def37fa1770fb56b53c735ea4" Sep 06 06:22:21.999719 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.999696 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm"] Sep 06 06:22:21.999818 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:22:21.999791 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10ecf1bc9dcba4db7f24c3a42ee0491688b4a4def37fa1770fb56b53c735ea4\": container with ID starting with d10ecf1bc9dcba4db7f24c3a42ee0491688b4a4def37fa1770fb56b53c735ea4 not found: ID does not exist" containerID="d10ecf1bc9dcba4db7f24c3a42ee0491688b4a4def37fa1770fb56b53c735ea4" Sep 06 06:22:21.999858 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.999815 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10ecf1bc9dcba4db7f24c3a42ee0491688b4a4def37fa1770fb56b53c735ea4"} err="failed to get container status \"d10ecf1bc9dcba4db7f24c3a42ee0491688b4a4def37fa1770fb56b53c735ea4\": rpc error: code = NotFound desc = could not find container \"d10ecf1bc9dcba4db7f24c3a42ee0491688b4a4def37fa1770fb56b53c735ea4\": container with ID starting with d10ecf1bc9dcba4db7f24c3a42ee0491688b4a4def37fa1770fb56b53c735ea4 not found: ID does not exist" Sep 06 06:22:21.999858 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:21.999835 2578 scope.go:117] "RemoveContainer" containerID="bfa28841c5d52be1b8f3c87ed1617d13dfe28a574e9e0252f4f7692eda6b796a" Sep 06 06:22:22.000137 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:22:22.000118 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa28841c5d52be1b8f3c87ed1617d13dfe28a574e9e0252f4f7692eda6b796a\": container with ID starting with bfa28841c5d52be1b8f3c87ed1617d13dfe28a574e9e0252f4f7692eda6b796a not found: ID does not exist" containerID="bfa28841c5d52be1b8f3c87ed1617d13dfe28a574e9e0252f4f7692eda6b796a" Sep 06 06:22:22.000198 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:22.000143 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa28841c5d52be1b8f3c87ed1617d13dfe28a574e9e0252f4f7692eda6b796a"} err="failed to get container status \"bfa28841c5d52be1b8f3c87ed1617d13dfe28a574e9e0252f4f7692eda6b796a\": rpc error: code = NotFound desc = could not find container \"bfa28841c5d52be1b8f3c87ed1617d13dfe28a574e9e0252f4f7692eda6b796a\": container with ID starting with bfa28841c5d52be1b8f3c87ed1617d13dfe28a574e9e0252f4f7692eda6b796a not found: ID does not exist" Sep 06 06:22:22.004412 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:22.004391 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ed85e8-openshift-default-75bb58b7d-dcmgm"] Sep 06 06:22:23.146082 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:23.146054 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" path="/var/lib/kubelet/pods/c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1/volumes" Sep 06 06:22:23.376042 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:23.376012 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:22:31.705477 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:31.705438 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr"] Sep 06 06:22:31.705873 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:31.705716 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" podUID="67934d5d-048f-4a3c-b02f-6f65d04c2fbb" containerName="maas-api" containerID="cri-o://7fa0c45c1d27a55fd725a8c96216e26912b9af96bf22f33ef52931ab3f0d43e0" gracePeriod=30 Sep 06 06:22:31.946539 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:31.946516 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:22:32.046012 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.045981 2578 generic.go:356] "Generic (PLEG): container finished" podID="67934d5d-048f-4a3c-b02f-6f65d04c2fbb" containerID="7fa0c45c1d27a55fd725a8c96216e26912b9af96bf22f33ef52931ab3f0d43e0" exitCode=0 Sep 06 06:22:32.046192 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.046030 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" Sep 06 06:22:32.046192 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.046041 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" event={"ID":"67934d5d-048f-4a3c-b02f-6f65d04c2fbb","Type":"ContainerDied","Data":"7fa0c45c1d27a55fd725a8c96216e26912b9af96bf22f33ef52931ab3f0d43e0"} Sep 06 06:22:32.046192 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.046091 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr" event={"ID":"67934d5d-048f-4a3c-b02f-6f65d04c2fbb","Type":"ContainerDied","Data":"541be4285de0ff519a673552c3376873b6230ae8e319cadb449ece12404aff3c"} Sep 06 06:22:32.046192 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.046115 2578 scope.go:117] "RemoveContainer" containerID="7fa0c45c1d27a55fd725a8c96216e26912b9af96bf22f33ef52931ab3f0d43e0" Sep 06 06:22:32.057283 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.057261 2578 scope.go:117] "RemoveContainer" containerID="7fa0c45c1d27a55fd725a8c96216e26912b9af96bf22f33ef52931ab3f0d43e0" Sep 06 06:22:32.057581 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:22:32.057559 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa0c45c1d27a55fd725a8c96216e26912b9af96bf22f33ef52931ab3f0d43e0\": container with ID starting with 7fa0c45c1d27a55fd725a8c96216e26912b9af96bf22f33ef52931ab3f0d43e0 not found: ID does not exist" containerID="7fa0c45c1d27a55fd725a8c96216e26912b9af96bf22f33ef52931ab3f0d43e0" Sep 06 06:22:32.057634 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.057590 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa0c45c1d27a55fd725a8c96216e26912b9af96bf22f33ef52931ab3f0d43e0"} err="failed to get container status \"7fa0c45c1d27a55fd725a8c96216e26912b9af96bf22f33ef52931ab3f0d43e0\": rpc error: code = NotFound desc = could not find container \"7fa0c45c1d27a55fd725a8c96216e26912b9af96bf22f33ef52931ab3f0d43e0\": container with ID starting with 7fa0c45c1d27a55fd725a8c96216e26912b9af96bf22f33ef52931ab3f0d43e0 not found: ID does not exist" Sep 06 06:22:32.065966 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.065940 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-maas-api-tls\") pod \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " Sep 06 06:22:32.066066 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.065996 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-openshift-service-ca\") pod \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " Sep 06 06:22:32.066066 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.066028 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5chrk\" (UniqueName: \"kubernetes.io/projected/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-kube-api-access-5chrk\") pod \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " Sep 06 06:22:32.066175 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.066147 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-metrics-certs\") pod \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\" (UID: \"67934d5d-048f-4a3c-b02f-6f65d04c2fbb\") " Sep 06 06:22:32.066385 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.066360 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67934d5d-048f-4a3c-b02f-6f65d04c2fbb" (UID: "67934d5d-048f-4a3c-b02f-6f65d04c2fbb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:22:32.066506 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.066490 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-openshift-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:32.068314 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.068292 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67934d5d-048f-4a3c-b02f-6f65d04c2fbb" (UID: "67934d5d-048f-4a3c-b02f-6f65d04c2fbb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:22:32.068421 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.068310 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "67934d5d-048f-4a3c-b02f-6f65d04c2fbb" (UID: "67934d5d-048f-4a3c-b02f-6f65d04c2fbb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:22:32.068421 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.068341 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-kube-api-access-5chrk" (OuterVolumeSpecName: "kube-api-access-5chrk") pod "67934d5d-048f-4a3c-b02f-6f65d04c2fbb" (UID: "67934d5d-048f-4a3c-b02f-6f65d04c2fbb"). InnerVolumeSpecName "kube-api-access-5chrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:22:32.167298 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.167259 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-maas-api-tls\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:32.167298 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.167299 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5chrk\" (UniqueName: \"kubernetes.io/projected/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-kube-api-access-5chrk\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:32.167562 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.167314 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67934d5d-048f-4a3c-b02f-6f65d04c2fbb-metrics-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:32.366594 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.366563 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr"] Sep 06 06:22:32.371967 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:32.371944 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c55e7-8df4ffc4-86npr"] Sep 06 06:22:33.144990 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:33.144959 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67934d5d-048f-4a3c-b02f-6f65d04c2fbb" path="/var/lib/kubelet/pods/67934d5d-048f-4a3c-b02f-6f65d04c2fbb/volumes" Sep 06 06:22:35.117829 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.117783 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" podUID="591c6655-156a-44c5-a59c-a20abe555780" containerName="main" containerID="cri-o://374dc1441d074c7090fddc157dedca944500d7020fe51407eb61f8aa6b17fd21" gracePeriod=45 Sep 06 06:22:35.377945 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.377873 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:22:35.503329 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.503295 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-home\") pod \"591c6655-156a-44c5-a59c-a20abe555780\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " Sep 06 06:22:35.503528 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.503362 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-model-cache\") pod \"591c6655-156a-44c5-a59c-a20abe555780\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " Sep 06 06:22:35.503528 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.503384 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-dshm\") pod \"591c6655-156a-44c5-a59c-a20abe555780\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " Sep 06 06:22:35.503528 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.503434 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591c6655-156a-44c5-a59c-a20abe555780-tls-certs\") pod \"591c6655-156a-44c5-a59c-a20abe555780\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " Sep 06 06:22:35.503528 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.503488 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-tmp-dir\") pod \"591c6655-156a-44c5-a59c-a20abe555780\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " Sep 06 06:22:35.503748 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.503544 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxbtf\" (UniqueName: \"kubernetes.io/projected/591c6655-156a-44c5-a59c-a20abe555780-kube-api-access-dxbtf\") pod \"591c6655-156a-44c5-a59c-a20abe555780\" (UID: \"591c6655-156a-44c5-a59c-a20abe555780\") " Sep 06 06:22:35.503748 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.503565 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-home" (OuterVolumeSpecName: "home") pod "591c6655-156a-44c5-a59c-a20abe555780" (UID: "591c6655-156a-44c5-a59c-a20abe555780"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:35.503748 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.503634 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-model-cache" (OuterVolumeSpecName: "model-cache") pod "591c6655-156a-44c5-a59c-a20abe555780" (UID: "591c6655-156a-44c5-a59c-a20abe555780"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:35.503860 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.503750 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "591c6655-156a-44c5-a59c-a20abe555780" (UID: "591c6655-156a-44c5-a59c-a20abe555780"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:35.503979 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.503961 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-home\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:35.504020 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.503983 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-model-cache\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:35.504020 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.503997 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-tmp-dir\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:35.505831 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.505800 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591c6655-156a-44c5-a59c-a20abe555780-kube-api-access-dxbtf" (OuterVolumeSpecName: "kube-api-access-dxbtf") pod "591c6655-156a-44c5-a59c-a20abe555780" (UID: "591c6655-156a-44c5-a59c-a20abe555780"). InnerVolumeSpecName "kube-api-access-dxbtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:22:35.506002 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.505829 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591c6655-156a-44c5-a59c-a20abe555780-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "591c6655-156a-44c5-a59c-a20abe555780" (UID: "591c6655-156a-44c5-a59c-a20abe555780"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:22:35.506002 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.505862 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-dshm" (OuterVolumeSpecName: "dshm") pod "591c6655-156a-44c5-a59c-a20abe555780" (UID: "591c6655-156a-44c5-a59c-a20abe555780"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:35.605056 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.605027 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591c6655-156a-44c5-a59c-a20abe555780-tls-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:35.605056 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.605050 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxbtf\" (UniqueName: \"kubernetes.io/projected/591c6655-156a-44c5-a59c-a20abe555780-kube-api-access-dxbtf\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:35.605056 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:35.605060 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/591c6655-156a-44c5-a59c-a20abe555780-dshm\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:36.079182 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:36.079147 2578 generic.go:356] "Generic (PLEG): container finished" podID="591c6655-156a-44c5-a59c-a20abe555780" containerID="374dc1441d074c7090fddc157dedca944500d7020fe51407eb61f8aa6b17fd21" exitCode=0 Sep 06 06:22:36.079411 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:36.079392 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" event={"ID":"591c6655-156a-44c5-a59c-a20abe555780","Type":"ContainerDied","Data":"374dc1441d074c7090fddc157dedca944500d7020fe51407eb61f8aa6b17fd21"} Sep 06 06:22:36.079460 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:36.079432 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" event={"ID":"591c6655-156a-44c5-a59c-a20abe555780","Type":"ContainerDied","Data":"010ceeaa7fdcdf26ff0e713f80ce174fcaa04820d714cbc7703390993d63d8ed"} Sep 06 06:22:36.079460 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:36.079436 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452" Sep 06 06:22:36.079559 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:36.079500 2578 scope.go:117] "RemoveContainer" containerID="374dc1441d074c7090fddc157dedca944500d7020fe51407eb61f8aa6b17fd21" Sep 06 06:22:36.089102 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:36.089084 2578 scope.go:117] "RemoveContainer" containerID="374dc1441d074c7090fddc157dedca944500d7020fe51407eb61f8aa6b17fd21" Sep 06 06:22:36.089343 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:22:36.089322 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374dc1441d074c7090fddc157dedca944500d7020fe51407eb61f8aa6b17fd21\": container with ID starting with 374dc1441d074c7090fddc157dedca944500d7020fe51407eb61f8aa6b17fd21 not found: ID does not exist" containerID="374dc1441d074c7090fddc157dedca944500d7020fe51407eb61f8aa6b17fd21" Sep 06 06:22:36.089393 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:36.089351 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374dc1441d074c7090fddc157dedca944500d7020fe51407eb61f8aa6b17fd21"} err="failed to get container status \"374dc1441d074c7090fddc157dedca944500d7020fe51407eb61f8aa6b17fd21\": rpc error: code = NotFound desc = could not find container \"374dc1441d074c7090fddc157dedca944500d7020fe51407eb61f8aa6b17fd21\": container with ID starting with 374dc1441d074c7090fddc157dedca944500d7020fe51407eb61f8aa6b17fd21 not found: ID does not exist" Sep 06 06:22:36.099365 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:36.099344 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452"] Sep 06 06:22:36.102533 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:36.102511 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-429831/sub-test-model-429831-kserve-69885f5c69-52452"] Sep 06 06:22:37.145073 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:37.145040 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="591c6655-156a-44c5-a59c-a20abe555780" path="/var/lib/kubelet/pods/591c6655-156a-44c5-a59c-a20abe555780/volumes" Sep 06 06:22:37.285879 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:37.285844 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:22:38.800740 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:38.800706 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d"] Sep 06 06:22:38.801290 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:38.801015 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" podUID="5e2fe0a4-ec00-4a32-bdfa-da789382de17" containerName="istio-proxy" containerID="cri-o://54f222113e8c715cbd59d0016a561a94878800bbd134fd644f72c2340783652a" gracePeriod=30 Sep 06 06:22:39.148051 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.147970 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45"] Sep 06 06:22:39.148355 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.148342 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591c6655-156a-44c5-a59c-a20abe555780" containerName="main" Sep 06 06:22:39.148398 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.148356 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="591c6655-156a-44c5-a59c-a20abe555780" containerName="main" Sep 06 06:22:39.148398 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.148368 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e893f445-e42e-450c-8934-01f038d21b19" containerName="mt-curl-7728-ef04a0" Sep 06 06:22:39.148398 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.148374 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e893f445-e42e-450c-8934-01f038d21b19" containerName="mt-curl-7728-ef04a0" Sep 06 06:22:39.148398 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.148385 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" containerName="istio-proxy" Sep 06 06:22:39.148398 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.148390 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" containerName="istio-proxy" Sep 06 06:22:39.148568 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.148405 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67934d5d-048f-4a3c-b02f-6f65d04c2fbb" containerName="maas-api" Sep 06 06:22:39.148568 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.148411 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67934d5d-048f-4a3c-b02f-6f65d04c2fbb" containerName="maas-api" Sep 06 06:22:39.148568 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.148420 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" containerName="istio-proxy" Sep 06 06:22:39.148568 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.148425 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" containerName="istio-proxy" Sep 06 06:22:39.148568 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.148483 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e893f445-e42e-450c-8934-01f038d21b19" containerName="mt-curl-7728-ef04a0" Sep 06 06:22:39.148568 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.148490 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" containerName="istio-proxy" Sep 06 06:22:39.148568 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.148499 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="591c6655-156a-44c5-a59c-a20abe555780" containerName="main" Sep 06 06:22:39.148568 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.148504 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c04fe53c-e5b0-4d5a-a064-f9ed623f6ed1" containerName="istio-proxy" Sep 06 06:22:39.148568 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.148510 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67934d5d-048f-4a3c-b02f-6f65d04c2fbb" containerName="maas-api" Sep 06 06:22:39.153143 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.153124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45" Sep 06 06:22:39.155544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.155514 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:22:39.155806 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.155777 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:22:39.158882 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.158858 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45"] Sep 06 06:22:39.242584 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.242554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e185790f-c7e8-49c3-8bf1-8e7674103e89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45\" (UID: \"e185790f-c7e8-49c3-8bf1-8e7674103e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45" Sep 06 06:22:39.343942 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.343897 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e185790f-c7e8-49c3-8bf1-8e7674103e89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45\" (UID: \"e185790f-c7e8-49c3-8bf1-8e7674103e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45" Sep 06 06:22:39.344560 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.344531 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e185790f-c7e8-49c3-8bf1-8e7674103e89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45\" (UID: \"e185790f-c7e8-49c3-8bf1-8e7674103e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45" Sep 06 06:22:39.466753 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.466659 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45" Sep 06 06:22:39.738703 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:39.738675 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45"] Sep 06 06:22:39.740458 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:22:39.740428 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode185790f_c7e8_49c3_8bf1_8e7674103e89.slice/crio-f05f0a6eafacdedda05783a528b5ca846676f7c1a7cb222ffb74aa443d7fda4f WatchSource:0}: Error finding container f05f0a6eafacdedda05783a528b5ca846676f7c1a7cb222ffb74aa443d7fda4f: Status 404 returned error can't find the container with id f05f0a6eafacdedda05783a528b5ca846676f7c1a7cb222ffb74aa443d7fda4f Sep 06 06:22:40.104195 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:40.104167 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45" event={"ID":"e185790f-c7e8-49c3-8bf1-8e7674103e89","Type":"ContainerStarted","Data":"61babb0a2bd1e9c4fccd0142ab7c5710260c75ba418c01a80d5f032111998900"} Sep 06 06:22:40.104557 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:40.104226 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45" event={"ID":"e185790f-c7e8-49c3-8bf1-8e7674103e89","Type":"ContainerStarted","Data":"f05f0a6eafacdedda05783a528b5ca846676f7c1a7cb222ffb74aa443d7fda4f"} Sep 06 06:22:40.118495 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:40.118451 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45" podStartSLOduration=1.118435643 podStartE2EDuration="1.118435643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:22:40.117199962 +0000 UTC m=+1725.515530441" watchObservedRunningTime="2026-09-06 06:22:40.118435643 +0000 UTC m=+1725.516766111" Sep 06 06:22:41.118612 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:41.118576 2578 generic.go:356] "Generic (PLEG): container finished" podID="e185790f-c7e8-49c3-8bf1-8e7674103e89" containerID="61babb0a2bd1e9c4fccd0142ab7c5710260c75ba418c01a80d5f032111998900" exitCode=0 Sep 06 06:22:41.119021 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:41.118653 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45" event={"ID":"e185790f-c7e8-49c3-8bf1-8e7674103e89","Type":"ContainerDied","Data":"61babb0a2bd1e9c4fccd0142ab7c5710260c75ba418c01a80d5f032111998900"} Sep 06 06:22:42.265550 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:42.265526 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45" Sep 06 06:22:42.375189 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:42.375098 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e185790f-c7e8-49c3-8bf1-8e7674103e89-maas-api-service-ca\") pod \"e185790f-c7e8-49c3-8bf1-8e7674103e89\" (UID: \"e185790f-c7e8-49c3-8bf1-8e7674103e89\") " Sep 06 06:22:42.375468 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:42.375444 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e185790f-c7e8-49c3-8bf1-8e7674103e89-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e185790f-c7e8-49c3-8bf1-8e7674103e89" (UID: "e185790f-c7e8-49c3-8bf1-8e7674103e89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:22:42.476536 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:42.476501 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e185790f-c7e8-49c3-8bf1-8e7674103e89-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:43.130692 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:43.130661 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45" Sep 06 06:22:43.130881 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:43.130700 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45" event={"ID":"e185790f-c7e8-49c3-8bf1-8e7674103e89","Type":"ContainerDied","Data":"f05f0a6eafacdedda05783a528b5ca846676f7c1a7cb222ffb74aa443d7fda4f"} Sep 06 06:22:43.130881 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:43.130746 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f05f0a6eafacdedda05783a528b5ca846676f7c1a7cb222ffb74aa443d7fda4f" Sep 06 06:22:44.078408 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.078342 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:22:44.138394 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.138359 2578 generic.go:356] "Generic (PLEG): container finished" podID="5e2fe0a4-ec00-4a32-bdfa-da789382de17" containerID="54f222113e8c715cbd59d0016a561a94878800bbd134fd644f72c2340783652a" exitCode=0 Sep 06 06:22:44.138577 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.138408 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" event={"ID":"5e2fe0a4-ec00-4a32-bdfa-da789382de17","Type":"ContainerDied","Data":"54f222113e8c715cbd59d0016a561a94878800bbd134fd644f72c2340783652a"} Sep 06 06:22:44.138577 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.138430 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" Sep 06 06:22:44.138577 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.138442 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d" event={"ID":"5e2fe0a4-ec00-4a32-bdfa-da789382de17","Type":"ContainerDied","Data":"22195abdfef9b5651376cee5979831fe9b4ffa329d54cafe30498988b8a2f200"} Sep 06 06:22:44.138577 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.138470 2578 scope.go:117] "RemoveContainer" containerID="54f222113e8c715cbd59d0016a561a94878800bbd134fd644f72c2340783652a" Sep 06 06:22:44.147805 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.147783 2578 scope.go:117] "RemoveContainer" containerID="89f8a454041d1aae5e1d9afb0e253f374734d4fc05e1dd23f4b3528b807cfb52" Sep 06 06:22:44.156375 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.156354 2578 scope.go:117] "RemoveContainer" containerID="54f222113e8c715cbd59d0016a561a94878800bbd134fd644f72c2340783652a" Sep 06 06:22:44.156665 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:22:44.156637 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f222113e8c715cbd59d0016a561a94878800bbd134fd644f72c2340783652a\": container with ID starting with 54f222113e8c715cbd59d0016a561a94878800bbd134fd644f72c2340783652a not found: ID does not exist" containerID="54f222113e8c715cbd59d0016a561a94878800bbd134fd644f72c2340783652a" Sep 06 06:22:44.156726 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.156674 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f222113e8c715cbd59d0016a561a94878800bbd134fd644f72c2340783652a"} err="failed to get container status \"54f222113e8c715cbd59d0016a561a94878800bbd134fd644f72c2340783652a\": rpc error: code = NotFound desc = could not find container \"54f222113e8c715cbd59d0016a561a94878800bbd134fd644f72c2340783652a\": container with ID starting with 54f222113e8c715cbd59d0016a561a94878800bbd134fd644f72c2340783652a not found: ID does not exist" Sep 06 06:22:44.156726 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.156691 2578 scope.go:117] "RemoveContainer" containerID="89f8a454041d1aae5e1d9afb0e253f374734d4fc05e1dd23f4b3528b807cfb52" Sep 06 06:22:44.157037 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:22:44.157015 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f8a454041d1aae5e1d9afb0e253f374734d4fc05e1dd23f4b3528b807cfb52\": container with ID starting with 89f8a454041d1aae5e1d9afb0e253f374734d4fc05e1dd23f4b3528b807cfb52 not found: ID does not exist" containerID="89f8a454041d1aae5e1d9afb0e253f374734d4fc05e1dd23f4b3528b807cfb52" Sep 06 06:22:44.157114 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.157047 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f8a454041d1aae5e1d9afb0e253f374734d4fc05e1dd23f4b3528b807cfb52"} err="failed to get container status \"89f8a454041d1aae5e1d9afb0e253f374734d4fc05e1dd23f4b3528b807cfb52\": rpc error: code = NotFound desc = could not find container \"89f8a454041d1aae5e1d9afb0e253f374734d4fc05e1dd23f4b3528b807cfb52\": container with ID starting with 89f8a454041d1aae5e1d9afb0e253f374734d4fc05e1dd23f4b3528b807cfb52 not found: ID does not exist" Sep 06 06:22:44.194376 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.194340 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-envoy\") pod \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " Sep 06 06:22:44.194544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.194400 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-token\") pod \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " Sep 06 06:22:44.194544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.194424 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-workload-certs\") pod \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " Sep 06 06:22:44.194544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.194480 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-workload-socket\") pod \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " Sep 06 06:22:44.194707 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.194571 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-data\") pod \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " Sep 06 06:22:44.194707 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.194595 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-credential-socket\") pod \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " Sep 06 06:22:44.194707 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.194626 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkjtw\" (UniqueName: \"kubernetes.io/projected/5e2fe0a4-ec00-4a32-bdfa-da789382de17-kube-api-access-pkjtw\") pod \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " Sep 06 06:22:44.194707 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.194665 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istiod-ca-cert\") pod \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " Sep 06 06:22:44.194707 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.194701 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-podinfo\") pod \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\" (UID: \"5e2fe0a4-ec00-4a32-bdfa-da789382de17\") " Sep 06 06:22:44.195012 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.194777 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e2fe0a4-ec00-4a32-bdfa-da789382de17" (UID: "5e2fe0a4-ec00-4a32-bdfa-da789382de17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:44.195012 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.194787 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e2fe0a4-ec00-4a32-bdfa-da789382de17" (UID: "5e2fe0a4-ec00-4a32-bdfa-da789382de17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:44.195124 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.195041 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e2fe0a4-ec00-4a32-bdfa-da789382de17" (UID: "5e2fe0a4-ec00-4a32-bdfa-da789382de17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:44.195124 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.195073 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e2fe0a4-ec00-4a32-bdfa-da789382de17" (UID: "5e2fe0a4-ec00-4a32-bdfa-da789382de17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:44.195227 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.195143 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:44.195227 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.195147 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e2fe0a4-ec00-4a32-bdfa-da789382de17" (UID: "5e2fe0a4-ec00-4a32-bdfa-da789382de17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:22:44.195227 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.195164 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:44.196865 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.196838 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e2fe0a4-ec00-4a32-bdfa-da789382de17" (UID: "5e2fe0a4-ec00-4a32-bdfa-da789382de17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:22:44.197065 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.196963 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e2fe0a4-ec00-4a32-bdfa-da789382de17" (UID: "5e2fe0a4-ec00-4a32-bdfa-da789382de17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:22:44.197398 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.197379 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2fe0a4-ec00-4a32-bdfa-da789382de17-kube-api-access-pkjtw" (OuterVolumeSpecName: "kube-api-access-pkjtw") pod "5e2fe0a4-ec00-4a32-bdfa-da789382de17" (UID: "5e2fe0a4-ec00-4a32-bdfa-da789382de17"). InnerVolumeSpecName "kube-api-access-pkjtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:22:44.197398 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.197388 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e2fe0a4-ec00-4a32-bdfa-da789382de17" (UID: "5e2fe0a4-ec00-4a32-bdfa-da789382de17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:22:44.296290 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.296252 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:44.296290 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.296286 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:44.296290 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.296295 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:44.296548 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.296307 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkjtw\" (UniqueName: \"kubernetes.io/projected/5e2fe0a4-ec00-4a32-bdfa-da789382de17-kube-api-access-pkjtw\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:44.296548 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.296316 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:44.296548 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.296323 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:44.296548 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.296332 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2fe0a4-ec00-4a32-bdfa-da789382de17-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:22:44.460170 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.460134 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d"] Sep 06 06:22:44.462126 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:44.462100 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1c55e7-openshift-default-7c76496c66-9sr2d"] Sep 06 06:22:45.146740 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:45.146708 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e2fe0a4-ec00-4a32-bdfa-da789382de17" path="/var/lib/kubelet/pods/5e2fe0a4-ec00-4a32-bdfa-da789382de17/volumes" Sep 06 06:22:49.789503 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:49.789464 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45"] Sep 06 06:22:49.791328 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:49.791305 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-340e00-xpr45"] Sep 06 06:22:51.145240 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:51.145197 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e185790f-c7e8-49c3-8bf1-8e7674103e89" path="/var/lib/kubelet/pods/e185790f-c7e8-49c3-8bf1-8e7674103e89/volumes" Sep 06 06:22:53.677865 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:53.677831 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:22:55.965975 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:55.965937 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg"] Sep 06 06:22:55.966454 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:55.966176 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" podUID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerName="istio-proxy" containerID="cri-o://20b627f666df6636620ae669816741872e1add453b5b9b0488d96ffa43c1048d" gracePeriod=30 Sep 06 06:22:56.296172 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.296137 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd"] Sep 06 06:22:56.296536 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.296523 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e2fe0a4-ec00-4a32-bdfa-da789382de17" containerName="istio-proxy" Sep 06 06:22:56.296588 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.296537 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e2fe0a4-ec00-4a32-bdfa-da789382de17" containerName="istio-proxy" Sep 06 06:22:56.296588 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.296555 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e2fe0a4-ec00-4a32-bdfa-da789382de17" containerName="istio-proxy" Sep 06 06:22:56.296588 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.296561 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e2fe0a4-ec00-4a32-bdfa-da789382de17" containerName="istio-proxy" Sep 06 06:22:56.296588 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.296567 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e185790f-c7e8-49c3-8bf1-8e7674103e89" containerName="revoke-keys" Sep 06 06:22:56.296588 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.296573 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e185790f-c7e8-49c3-8bf1-8e7674103e89" containerName="revoke-keys" Sep 06 06:22:56.296730 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.296655 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e2fe0a4-ec00-4a32-bdfa-da789382de17" containerName="istio-proxy" Sep 06 06:22:56.296730 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.296661 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e185790f-c7e8-49c3-8bf1-8e7674103e89" containerName="revoke-keys" Sep 06 06:22:56.301035 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.301015 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd" Sep 06 06:22:56.303300 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.303272 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:22:56.303615 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.303592 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:22:56.305609 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.305587 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd"] Sep 06 06:22:56.414362 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.414326 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac1257e-c07d-49ad-a35f-b8cafc8bbdc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd\" (UID: \"aac1257e-c07d-49ad-a35f-b8cafc8bbdc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd" Sep 06 06:22:56.515699 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.515656 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac1257e-c07d-49ad-a35f-b8cafc8bbdc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd\" (UID: \"aac1257e-c07d-49ad-a35f-b8cafc8bbdc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd" Sep 06 06:22:56.516326 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.516305 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac1257e-c07d-49ad-a35f-b8cafc8bbdc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd\" (UID: \"aac1257e-c07d-49ad-a35f-b8cafc8bbdc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd" Sep 06 06:22:56.613821 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.613707 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd" Sep 06 06:22:56.885409 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:56.885371 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd"] Sep 06 06:22:56.887169 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:22:56.887141 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac1257e_c07d_49ad_a35f_b8cafc8bbdc8.slice/crio-5201fae973b7b58ef09a964b46376554a68d0070a1e0e7e4a4368c62012d9a0d WatchSource:0}: Error finding container 5201fae973b7b58ef09a964b46376554a68d0070a1e0e7e4a4368c62012d9a0d: Status 404 returned error can't find the container with id 5201fae973b7b58ef09a964b46376554a68d0070a1e0e7e4a4368c62012d9a0d Sep 06 06:22:57.214841 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:57.214749 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd" event={"ID":"aac1257e-c07d-49ad-a35f-b8cafc8bbdc8","Type":"ContainerStarted","Data":"ccc5a6e9c8e707ebe6ff516a1af1362e1dc8052dce12d2509910ed328d65f7f7"} Sep 06 06:22:57.214841 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:57.214798 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd" event={"ID":"aac1257e-c07d-49ad-a35f-b8cafc8bbdc8","Type":"ContainerStarted","Data":"5201fae973b7b58ef09a964b46376554a68d0070a1e0e7e4a4368c62012d9a0d"} Sep 06 06:22:57.230429 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:57.230377 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd" podStartSLOduration=1.230362292 podStartE2EDuration="1.230362292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:22:57.22954936 +0000 UTC m=+1742.627879832" watchObservedRunningTime="2026-09-06 06:22:57.230362292 +0000 UTC m=+1742.628692759" Sep 06 06:22:58.228510 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:58.228466 2578 generic.go:356] "Generic (PLEG): container finished" podID="aac1257e-c07d-49ad-a35f-b8cafc8bbdc8" containerID="ccc5a6e9c8e707ebe6ff516a1af1362e1dc8052dce12d2509910ed328d65f7f7" exitCode=0 Sep 06 06:22:58.229285 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:58.228528 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd" event={"ID":"aac1257e-c07d-49ad-a35f-b8cafc8bbdc8","Type":"ContainerDied","Data":"ccc5a6e9c8e707ebe6ff516a1af1362e1dc8052dce12d2509910ed328d65f7f7"} Sep 06 06:22:59.377615 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:59.377583 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd" Sep 06 06:22:59.548034 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:59.547998 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac1257e-c07d-49ad-a35f-b8cafc8bbdc8-maas-api-service-ca\") pod \"aac1257e-c07d-49ad-a35f-b8cafc8bbdc8\" (UID: \"aac1257e-c07d-49ad-a35f-b8cafc8bbdc8\") " Sep 06 06:22:59.548344 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:59.548317 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac1257e-c07d-49ad-a35f-b8cafc8bbdc8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aac1257e-c07d-49ad-a35f-b8cafc8bbdc8" (UID: "aac1257e-c07d-49ad-a35f-b8cafc8bbdc8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:22:59.649181 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:22:59.649141 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac1257e-c07d-49ad-a35f-b8cafc8bbdc8-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:00.240914 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:00.240879 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd" Sep 06 06:23:00.241096 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:00.240883 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd" event={"ID":"aac1257e-c07d-49ad-a35f-b8cafc8bbdc8","Type":"ContainerDied","Data":"5201fae973b7b58ef09a964b46376554a68d0070a1e0e7e4a4368c62012d9a0d"} Sep 06 06:23:00.241096 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:00.241026 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5201fae973b7b58ef09a964b46376554a68d0070a1e0e7e4a4368c62012d9a0d" Sep 06 06:23:01.223573 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.223549 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:23:01.249123 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.249049 2578 generic.go:356] "Generic (PLEG): container finished" podID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerID="20b627f666df6636620ae669816741872e1add453b5b9b0488d96ffa43c1048d" exitCode=0 Sep 06 06:23:01.249123 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.249085 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" event={"ID":"e72e2000-5d2c-488d-ba88-9e23c8eee0c7","Type":"ContainerDied","Data":"20b627f666df6636620ae669816741872e1add453b5b9b0488d96ffa43c1048d"} Sep 06 06:23:01.249123 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.249113 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" Sep 06 06:23:01.249443 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.249149 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg" event={"ID":"e72e2000-5d2c-488d-ba88-9e23c8eee0c7","Type":"ContainerDied","Data":"f4edb02c711203115a638900f50ef7c19c553d629bec386a1507cb3cb1f4d40b"} Sep 06 06:23:01.249443 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.249182 2578 scope.go:117] "RemoveContainer" containerID="20b627f666df6636620ae669816741872e1add453b5b9b0488d96ffa43c1048d" Sep 06 06:23:01.260643 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.260622 2578 scope.go:117] "RemoveContainer" containerID="0c774ee4b3b4ad242006540339b9862938f1791c928426ee5d718af382e4959e" Sep 06 06:23:01.262931 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.262883 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfzrq\" (UniqueName: \"kubernetes.io/projected/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-kube-api-access-sfzrq\") pod \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " Sep 06 06:23:01.262931 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.262932 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-workload-certs\") pod \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " Sep 06 06:23:01.263106 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.263041 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-data\") pod \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " Sep 06 06:23:01.263106 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.263084 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istiod-ca-cert\") pod \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " Sep 06 06:23:01.263220 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.263118 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-workload-socket\") pod \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " Sep 06 06:23:01.263220 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.263150 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-envoy\") pod \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " Sep 06 06:23:01.263220 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.263183 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-credential-socket\") pod \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " Sep 06 06:23:01.263220 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.263201 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e72e2000-5d2c-488d-ba88-9e23c8eee0c7" (UID: "e72e2000-5d2c-488d-ba88-9e23c8eee0c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:23:01.263430 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.263230 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-podinfo\") pod \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " Sep 06 06:23:01.263430 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.263279 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-data" (OuterVolumeSpecName: "istio-data") pod "e72e2000-5d2c-488d-ba88-9e23c8eee0c7" (UID: "e72e2000-5d2c-488d-ba88-9e23c8eee0c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:23:01.263430 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.263319 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-token\") pod \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\" (UID: \"e72e2000-5d2c-488d-ba88-9e23c8eee0c7\") " Sep 06 06:23:01.263752 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.263440 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e72e2000-5d2c-488d-ba88-9e23c8eee0c7" (UID: "e72e2000-5d2c-488d-ba88-9e23c8eee0c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:23:01.263752 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.263507 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e72e2000-5d2c-488d-ba88-9e23c8eee0c7" (UID: "e72e2000-5d2c-488d-ba88-9e23c8eee0c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:23:01.263752 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.263714 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:01.263752 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.263734 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:01.263752 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.263747 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:01.264101 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.263760 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:01.264101 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.264050 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e72e2000-5d2c-488d-ba88-9e23c8eee0c7" (UID: "e72e2000-5d2c-488d-ba88-9e23c8eee0c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:23:01.265332 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.265286 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-kube-api-access-sfzrq" (OuterVolumeSpecName: "kube-api-access-sfzrq") pod "e72e2000-5d2c-488d-ba88-9e23c8eee0c7" (UID: "e72e2000-5d2c-488d-ba88-9e23c8eee0c7"). InnerVolumeSpecName "kube-api-access-sfzrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:23:01.265763 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.265734 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e72e2000-5d2c-488d-ba88-9e23c8eee0c7" (UID: "e72e2000-5d2c-488d-ba88-9e23c8eee0c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:23:01.265869 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.265850 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e72e2000-5d2c-488d-ba88-9e23c8eee0c7" (UID: "e72e2000-5d2c-488d-ba88-9e23c8eee0c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:23:01.265985 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.265969 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "e72e2000-5d2c-488d-ba88-9e23c8eee0c7" (UID: "e72e2000-5d2c-488d-ba88-9e23c8eee0c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:23:01.285105 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.285083 2578 scope.go:117] "RemoveContainer" containerID="20b627f666df6636620ae669816741872e1add453b5b9b0488d96ffa43c1048d" Sep 06 06:23:01.285463 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:23:01.285440 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b627f666df6636620ae669816741872e1add453b5b9b0488d96ffa43c1048d\": container with ID starting with 20b627f666df6636620ae669816741872e1add453b5b9b0488d96ffa43c1048d not found: ID does not exist" containerID="20b627f666df6636620ae669816741872e1add453b5b9b0488d96ffa43c1048d" Sep 06 06:23:01.285514 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.285471 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b627f666df6636620ae669816741872e1add453b5b9b0488d96ffa43c1048d"} err="failed to get container status \"20b627f666df6636620ae669816741872e1add453b5b9b0488d96ffa43c1048d\": rpc error: code = NotFound desc = could not find container \"20b627f666df6636620ae669816741872e1add453b5b9b0488d96ffa43c1048d\": container with ID starting with 20b627f666df6636620ae669816741872e1add453b5b9b0488d96ffa43c1048d not found: ID does not exist" Sep 06 06:23:01.285514 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.285489 2578 scope.go:117] "RemoveContainer" containerID="0c774ee4b3b4ad242006540339b9862938f1791c928426ee5d718af382e4959e" Sep 06 06:23:01.285739 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:23:01.285722 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c774ee4b3b4ad242006540339b9862938f1791c928426ee5d718af382e4959e\": container with ID starting with 0c774ee4b3b4ad242006540339b9862938f1791c928426ee5d718af382e4959e not found: ID does not exist" containerID="0c774ee4b3b4ad242006540339b9862938f1791c928426ee5d718af382e4959e" Sep 06 06:23:01.285786 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.285742 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c774ee4b3b4ad242006540339b9862938f1791c928426ee5d718af382e4959e"} err="failed to get container status \"0c774ee4b3b4ad242006540339b9862938f1791c928426ee5d718af382e4959e\": rpc error: code = NotFound desc = could not find container \"0c774ee4b3b4ad242006540339b9862938f1791c928426ee5d718af382e4959e\": container with ID starting with 0c774ee4b3b4ad242006540339b9862938f1791c928426ee5d718af382e4959e not found: ID does not exist" Sep 06 06:23:01.365027 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.364988 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:01.365027 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.365021 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfzrq\" (UniqueName: \"kubernetes.io/projected/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-kube-api-access-sfzrq\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:01.365027 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.365034 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:01.365271 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.365046 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:01.365271 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.365059 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72e2000-5d2c-488d-ba88-9e23c8eee0c7-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:01.570783 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.570749 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg"] Sep 06 06:23:01.574658 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:01.574634 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-340e00-openshift-default-5586fbccfd-xl5cg"] Sep 06 06:23:03.145089 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:03.145058 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" path="/var/lib/kubelet/pods/e72e2000-5d2c-488d-ba88-9e23c8eee0c7/volumes" Sep 06 06:23:03.881395 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:03.881359 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:23:06.914356 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:06.914320 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd"] Sep 06 06:23:06.917077 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:06.917050 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b7c728-hkvjd"] Sep 06 06:23:07.144997 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:07.144950 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aac1257e-c07d-49ad-a35f-b8cafc8bbdc8" path="/var/lib/kubelet/pods/aac1257e-c07d-49ad-a35f-b8cafc8bbdc8/volumes" Sep 06 06:23:13.286584 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:13.286539 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29"] Sep 06 06:23:13.287200 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:13.286825 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" podUID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" containerName="istio-proxy" containerID="cri-o://3bfa94432733935b578b82719fb642ef5ba1eb44ddc64166a339ad79c81ffdd7" gracePeriod=30 Sep 06 06:23:17.186150 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:17.186110 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:23:18.538149 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.538120 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:23:18.627786 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.627696 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-envoy\") pod \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " Sep 06 06:23:18.627786 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.627737 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms7dc\" (UniqueName: \"kubernetes.io/projected/1d1a22d3-db9b-48ec-bd59-6067b4086dec-kube-api-access-ms7dc\") pod \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " Sep 06 06:23:18.627786 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.627767 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istiod-ca-cert\") pod \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " Sep 06 06:23:18.628136 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.627805 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-data\") pod \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " Sep 06 06:23:18.628136 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.627865 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-token\") pod \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " Sep 06 06:23:18.628136 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.627898 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-credential-socket\") pod \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " Sep 06 06:23:18.628136 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.627960 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-workload-certs\") pod \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " Sep 06 06:23:18.628136 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.627995 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-workload-socket\") pod \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " Sep 06 06:23:18.628136 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.628035 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-podinfo\") pod \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\" (UID: \"1d1a22d3-db9b-48ec-bd59-6067b4086dec\") " Sep 06 06:23:18.628442 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.628193 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-data" (OuterVolumeSpecName: "istio-data") pod "1d1a22d3-db9b-48ec-bd59-6067b4086dec" (UID: "1d1a22d3-db9b-48ec-bd59-6067b4086dec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:23:18.628442 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.628218 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1d1a22d3-db9b-48ec-bd59-6067b4086dec" (UID: "1d1a22d3-db9b-48ec-bd59-6067b4086dec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:23:18.628723 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.628448 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:18.628723 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.628469 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:18.628723 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.628481 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1d1a22d3-db9b-48ec-bd59-6067b4086dec" (UID: "1d1a22d3-db9b-48ec-bd59-6067b4086dec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:23:18.628723 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.628563 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1d1a22d3-db9b-48ec-bd59-6067b4086dec" (UID: "1d1a22d3-db9b-48ec-bd59-6067b4086dec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:23:18.628723 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.628645 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1d1a22d3-db9b-48ec-bd59-6067b4086dec" (UID: "1d1a22d3-db9b-48ec-bd59-6067b4086dec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:23:18.630294 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.630265 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1d1a22d3-db9b-48ec-bd59-6067b4086dec" (UID: "1d1a22d3-db9b-48ec-bd59-6067b4086dec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:23:18.630429 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.630306 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-token" (OuterVolumeSpecName: "istio-token") pod "1d1a22d3-db9b-48ec-bd59-6067b4086dec" (UID: "1d1a22d3-db9b-48ec-bd59-6067b4086dec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:23:18.630429 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.630351 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1a22d3-db9b-48ec-bd59-6067b4086dec-kube-api-access-ms7dc" (OuterVolumeSpecName: "kube-api-access-ms7dc") pod "1d1a22d3-db9b-48ec-bd59-6067b4086dec" (UID: "1d1a22d3-db9b-48ec-bd59-6067b4086dec"). InnerVolumeSpecName "kube-api-access-ms7dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:23:18.630522 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.630483 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1d1a22d3-db9b-48ec-bd59-6067b4086dec" (UID: "1d1a22d3-db9b-48ec-bd59-6067b4086dec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:23:18.729767 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.729730 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:18.729767 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.729762 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:18.729767 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.729772 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:18.729767 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.729780 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:18.730105 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.729788 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:18.730105 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.729796 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d1a22d3-db9b-48ec-bd59-6067b4086dec-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:18.730105 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:18.729804 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms7dc\" (UniqueName: \"kubernetes.io/projected/1d1a22d3-db9b-48ec-bd59-6067b4086dec-kube-api-access-ms7dc\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:23:19.356288 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:19.356252 2578 generic.go:356] "Generic (PLEG): container finished" podID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" containerID="3bfa94432733935b578b82719fb642ef5ba1eb44ddc64166a339ad79c81ffdd7" exitCode=0 Sep 06 06:23:19.356476 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:19.356300 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" Sep 06 06:23:19.356476 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:19.356336 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" event={"ID":"1d1a22d3-db9b-48ec-bd59-6067b4086dec","Type":"ContainerDied","Data":"3bfa94432733935b578b82719fb642ef5ba1eb44ddc64166a339ad79c81ffdd7"} Sep 06 06:23:19.356476 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:19.356392 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29" event={"ID":"1d1a22d3-db9b-48ec-bd59-6067b4086dec","Type":"ContainerDied","Data":"daf28eb2b49d5c617cc3b45ad79cc6f49af38bf63a9c9d2099ee9fa9d25cbc2d"} Sep 06 06:23:19.356476 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:19.356416 2578 scope.go:117] "RemoveContainer" containerID="3bfa94432733935b578b82719fb642ef5ba1eb44ddc64166a339ad79c81ffdd7" Sep 06 06:23:19.365849 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:19.365832 2578 scope.go:117] "RemoveContainer" containerID="148f919d44d8463c66ae8b198af3d13e004f251573f6f1c40336065c82ff4da4" Sep 06 06:23:19.372834 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:19.372814 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29"] Sep 06 06:23:19.374831 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:19.374815 2578 scope.go:117] "RemoveContainer" containerID="3bfa94432733935b578b82719fb642ef5ba1eb44ddc64166a339ad79c81ffdd7" Sep 06 06:23:19.375113 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:23:19.375093 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bfa94432733935b578b82719fb642ef5ba1eb44ddc64166a339ad79c81ffdd7\": container with ID starting with 3bfa94432733935b578b82719fb642ef5ba1eb44ddc64166a339ad79c81ffdd7 not found: ID does not exist" containerID="3bfa94432733935b578b82719fb642ef5ba1eb44ddc64166a339ad79c81ffdd7" Sep 06 06:23:19.375158 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:19.375122 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfa94432733935b578b82719fb642ef5ba1eb44ddc64166a339ad79c81ffdd7"} err="failed to get container status \"3bfa94432733935b578b82719fb642ef5ba1eb44ddc64166a339ad79c81ffdd7\": rpc error: code = NotFound desc = could not find container \"3bfa94432733935b578b82719fb642ef5ba1eb44ddc64166a339ad79c81ffdd7\": container with ID starting with 3bfa94432733935b578b82719fb642ef5ba1eb44ddc64166a339ad79c81ffdd7 not found: ID does not exist" Sep 06 06:23:19.375158 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:19.375146 2578 scope.go:117] "RemoveContainer" containerID="148f919d44d8463c66ae8b198af3d13e004f251573f6f1c40336065c82ff4da4" Sep 06 06:23:19.375438 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:23:19.375420 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"148f919d44d8463c66ae8b198af3d13e004f251573f6f1c40336065c82ff4da4\": container with ID starting with 148f919d44d8463c66ae8b198af3d13e004f251573f6f1c40336065c82ff4da4 not found: ID does not exist" containerID="148f919d44d8463c66ae8b198af3d13e004f251573f6f1c40336065c82ff4da4" Sep 06 06:23:19.375479 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:19.375449 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"148f919d44d8463c66ae8b198af3d13e004f251573f6f1c40336065c82ff4da4"} err="failed to get container status \"148f919d44d8463c66ae8b198af3d13e004f251573f6f1c40336065c82ff4da4\": rpc error: code = NotFound desc = could not find container \"148f919d44d8463c66ae8b198af3d13e004f251573f6f1c40336065c82ff4da4\": container with ID starting with 148f919d44d8463c66ae8b198af3d13e004f251573f6f1c40336065c82ff4da4 not found: ID does not exist" Sep 06 06:23:19.377829 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:19.377808 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b7c728-openshift-default-7f9b8cf766-xxc29"] Sep 06 06:23:21.145718 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:21.145685 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" path="/var/lib/kubelet/pods/1d1a22d3-db9b-48ec-bd59-6067b4086dec/volumes" Sep 06 06:23:27.382749 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:27.382708 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:23:42.680307 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:42.680272 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:23:48.114488 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.114403 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6"] Sep 06 06:23:48.115032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.114766 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" containerName="istio-proxy" Sep 06 06:23:48.115032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.114781 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" containerName="istio-proxy" Sep 06 06:23:48.115032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.114798 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerName="istio-proxy" Sep 06 06:23:48.115032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.114804 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerName="istio-proxy" Sep 06 06:23:48.115032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.114810 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" containerName="istio-proxy" Sep 06 06:23:48.115032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.114816 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" containerName="istio-proxy" Sep 06 06:23:48.115032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.114830 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac1257e-c07d-49ad-a35f-b8cafc8bbdc8" containerName="revoke-keys" Sep 06 06:23:48.115032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.114834 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac1257e-c07d-49ad-a35f-b8cafc8bbdc8" containerName="revoke-keys" Sep 06 06:23:48.115032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.114894 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerName="istio-proxy" Sep 06 06:23:48.115032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.114920 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e2fe0a4-ec00-4a32-bdfa-da789382de17" containerName="istio-proxy" Sep 06 06:23:48.115032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.114929 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac1257e-c07d-49ad-a35f-b8cafc8bbdc8" containerName="revoke-keys" Sep 06 06:23:48.115032 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.114939 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" containerName="istio-proxy" Sep 06 06:23:48.118190 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.118166 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.120617 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.120596 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a36847-openshift-default-dockercfg-rjgw6\"" Sep 06 06:23:48.129059 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.129035 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6"] Sep 06 06:23:48.215658 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.215622 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c9849b-a620-484d-a4e7-627b706e2b01-istio-token\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.215825 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.215669 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k5tw\" (UniqueName: \"kubernetes.io/projected/34c9849b-a620-484d-a4e7-627b706e2b01-kube-api-access-2k5tw\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.215825 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.215741 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c9849b-a620-484d-a4e7-627b706e2b01-istiod-ca-cert\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.215825 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.215767 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-workload-certs\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.215825 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.215815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-credential-socket\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.216014 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.215886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-workload-socket\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.216014 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.215959 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-istio-envoy\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.216014 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.215986 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c9849b-a620-484d-a4e7-627b706e2b01-istio-podinfo\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.216014 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.216005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-istio-data\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.317054 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.317008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-workload-socket\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.317274 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.317074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-istio-envoy\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.317274 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.317107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c9849b-a620-484d-a4e7-627b706e2b01-istio-podinfo\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.317274 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.317130 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-istio-data\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.317668 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.317637 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-workload-socket\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.317985 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.317681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c9849b-a620-484d-a4e7-627b706e2b01-istio-token\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.318301 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.318240 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k5tw\" (UniqueName: \"kubernetes.io/projected/34c9849b-a620-484d-a4e7-627b706e2b01-kube-api-access-2k5tw\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.318407 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.318368 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-istio-data\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.318571 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.318464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c9849b-a620-484d-a4e7-627b706e2b01-istiod-ca-cert\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.318796 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.318771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-workload-certs\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.318891 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.318868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-credential-socket\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.319110 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.319089 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-workload-certs\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.319193 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.319143 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-credential-socket\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.319238 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.319219 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c9849b-a620-484d-a4e7-627b706e2b01-istiod-ca-cert\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.320114 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.320092 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-istio-envoy\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.320272 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.320254 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c9849b-a620-484d-a4e7-627b706e2b01-istio-podinfo\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.324765 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.324743 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c9849b-a620-484d-a4e7-627b706e2b01-istio-token\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.325124 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.325104 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k5tw\" (UniqueName: \"kubernetes.io/projected/34c9849b-a620-484d-a4e7-627b706e2b01-kube-api-access-2k5tw\") pod \"e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.428962 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.428826 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:48.703011 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.702981 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6"] Sep 06 06:23:48.705229 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:23:48.705197 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c9849b_a620_484d_a4e7_627b706e2b01.slice/crio-4c3a7b583e0a7f96aa28c2339b23a1e9a641f4f281c58caae45aca273d2c2053 WatchSource:0}: Error finding container 4c3a7b583e0a7f96aa28c2339b23a1e9a641f4f281c58caae45aca273d2c2053: Status 404 returned error can't find the container with id 4c3a7b583e0a7f96aa28c2339b23a1e9a641f4f281c58caae45aca273d2c2053 Sep 06 06:23:48.708046 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.708006 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:23:48.708127 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.708099 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:23:48.708173 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:48.708147 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:23:49.519473 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:49.519438 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" event={"ID":"34c9849b-a620-484d-a4e7-627b706e2b01","Type":"ContainerStarted","Data":"588fcde7a52bcda484a350479349d7670632fab7ef014b8a2e77c0c059bb3039"} Sep 06 06:23:49.519839 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:49.519493 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" event={"ID":"34c9849b-a620-484d-a4e7-627b706e2b01","Type":"ContainerStarted","Data":"4c3a7b583e0a7f96aa28c2339b23a1e9a641f4f281c58caae45aca273d2c2053"} Sep 06 06:23:49.537866 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:49.537817 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" podStartSLOduration=1.5378014119999999 podStartE2EDuration="1.537801412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:23:49.536155137 +0000 UTC m=+1794.934485605" watchObservedRunningTime="2026-09-06 06:23:49.537801412 +0000 UTC m=+1794.936131928" Sep 06 06:23:50.429235 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:50.429194 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:50.434364 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:50.434334 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:50.525207 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:50.525174 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:50.526184 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:50.526165 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:23:52.784663 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.784626 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn"] Sep 06 06:23:52.785069 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.785006 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerName="istio-proxy" Sep 06 06:23:52.785069 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.785018 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerName="istio-proxy" Sep 06 06:23:52.785160 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.785099 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d1a22d3-db9b-48ec-bd59-6067b4086dec" containerName="istio-proxy" Sep 06 06:23:52.785160 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.785115 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e72e2000-5d2c-488d-ba88-9e23c8eee0c7" containerName="istio-proxy" Sep 06 06:23:52.788583 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.788567 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:52.790563 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.790539 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a36847-serving-cert\"" Sep 06 06:23:52.790563 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.790550 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5vd8t\"" Sep 06 06:23:52.790744 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.790594 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 06 06:23:52.790744 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.790611 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 06 06:23:52.790744 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.790554 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:23:52.798837 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.798806 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn"] Sep 06 06:23:52.863005 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.862970 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bafea7-a0de-4229-b009-cee4614b0afb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:52.863234 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.863020 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-metrics-certs\") pod \"maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:52.863234 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.863097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:52.863234 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.863181 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk9jj\" (UniqueName: \"kubernetes.io/projected/c2bafea7-a0de-4229-b009-cee4614b0afb-kube-api-access-wk9jj\") pod \"maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:52.963932 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.963876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk9jj\" (UniqueName: \"kubernetes.io/projected/c2bafea7-a0de-4229-b009-cee4614b0afb-kube-api-access-wk9jj\") pod \"maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:52.964128 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.963966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bafea7-a0de-4229-b009-cee4614b0afb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:52.964128 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.963999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-metrics-certs\") pod \"maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:52.964128 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.964041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:52.964339 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:23:52.964163 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-serving-cert: secret "maas-api-e2e-ipp-a-a36847-serving-cert" not found Sep 06 06:23:52.964339 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:23:52.964233 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-maas-api-tls podName:c2bafea7-a0de-4229-b009-cee4614b0afb nodeName:}" failed. No retries permitted until 2026-09-06 06:23:53.464217673 +0000 UTC m=+1798.862548119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-maas-api-tls") pod "maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" (UID: "c2bafea7-a0de-4229-b009-cee4614b0afb") : secret "maas-api-e2e-ipp-a-a36847-serving-cert" not found Sep 06 06:23:52.964620 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.964598 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bafea7-a0de-4229-b009-cee4614b0afb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:52.966767 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.966738 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-metrics-certs\") pod \"maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:52.974496 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:52.974471 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk9jj\" (UniqueName: \"kubernetes.io/projected/c2bafea7-a0de-4229-b009-cee4614b0afb-kube-api-access-wk9jj\") pod \"maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:53.470291 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:53.470251 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:53.470472 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:23:53.470416 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-serving-cert: secret "maas-api-e2e-ipp-a-a36847-serving-cert" not found Sep 06 06:23:53.470517 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:23:53.470484 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-maas-api-tls podName:c2bafea7-a0de-4229-b009-cee4614b0afb nodeName:}" failed. No retries permitted until 2026-09-06 06:23:54.470468512 +0000 UTC m=+1799.868798962 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-maas-api-tls") pod "maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" (UID: "c2bafea7-a0de-4229-b009-cee4614b0afb") : secret "maas-api-e2e-ipp-a-a36847-serving-cert" not found Sep 06 06:23:54.480572 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:54.480529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:54.483239 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:54.483216 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:54.603368 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:54.603324 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:54.875794 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:54.875769 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn"] Sep 06 06:23:54.877891 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:23:54.877846 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2bafea7_a0de_4229_b009_cee4614b0afb.slice/crio-191d02190aa286bdd2645f6bd0b97b90c997351d3456ed8008564aa319147350 WatchSource:0}: Error finding container 191d02190aa286bdd2645f6bd0b97b90c997351d3456ed8008564aa319147350: Status 404 returned error can't find the container with id 191d02190aa286bdd2645f6bd0b97b90c997351d3456ed8008564aa319147350 Sep 06 06:23:55.200670 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:55.200582 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 06:23:55.203520 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:55.203498 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 06:23:55.536252 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:55.536220 2578 scope.go:117] "RemoveContainer" containerID="b16698eeb6c0825d8d1bcc6c0b1d09f6895ce1f8e53e95249e07da21b9c12f67" Sep 06 06:23:55.545157 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:55.545137 2578 scope.go:117] "RemoveContainer" containerID="2a377082283b2c666eb9335801fde1ec87eb920e2ca018ab700d484ed6869648" Sep 06 06:23:55.555198 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:55.555173 2578 scope.go:117] "RemoveContainer" containerID="23bef4cdd197a6f9f2449ab60c5c7fa287704b541856726b069faf8d83b8b2dd" Sep 06 06:23:55.556335 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:55.556302 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" event={"ID":"c2bafea7-a0de-4229-b009-cee4614b0afb","Type":"ContainerStarted","Data":"7467c641ad4ea318b4dfb9f2ad33ea3b80717569d6108cd4951ed79eb0832504"} Sep 06 06:23:55.556605 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:55.556362 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" event={"ID":"c2bafea7-a0de-4229-b009-cee4614b0afb","Type":"ContainerStarted","Data":"191d02190aa286bdd2645f6bd0b97b90c997351d3456ed8008564aa319147350"} Sep 06 06:23:55.556605 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:55.556404 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:23:55.563330 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:55.563310 2578 scope.go:117] "RemoveContainer" containerID="7872100d80caf279a03c65268fa0102843bbf622c4deb50ff152f2919190d88c" Sep 06 06:23:55.572713 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:55.572668 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" podStartSLOduration=3.168140926 podStartE2EDuration="3.572652945s" totalImagesPullingTime="404.512019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:23:55.570334487 +0000 UTC m=+1800.968664955" watchObservedRunningTime="2026-09-06 06:23:55.572652945 +0000 UTC m=+1800.970983413" Sep 06 06:23:56.482615 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:23:56.482575 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:24:01.576617 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:01.576586 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:24:06.161547 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:06.160407 2578 patch_prober.go:28] interesting pod/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": read tcp 10.133.0.2:57946->10.133.0.81:15021: read: connection reset by peer" start-of-body= Sep 06 06:24:06.161547 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:06.160485 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" podUID="34c9849b-a620-484d-a4e7-627b706e2b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": read tcp 10.133.0.2:57946->10.133.0.81:15021: read: connection reset by peer" Sep 06 06:24:06.621835 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:06.621800 2578 generic.go:356] "Generic (PLEG): container finished" podID="34c9849b-a620-484d-a4e7-627b706e2b01" containerID="588fcde7a52bcda484a350479349d7670632fab7ef014b8a2e77c0c059bb3039" exitCode=0 Sep 06 06:24:06.622044 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:06.621874 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" event={"ID":"34c9849b-a620-484d-a4e7-627b706e2b01","Type":"ContainerDied","Data":"588fcde7a52bcda484a350479349d7670632fab7ef014b8a2e77c0c059bb3039"} Sep 06 06:24:06.622329 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:06.622313 2578 scope.go:117] "RemoveContainer" containerID="588fcde7a52bcda484a350479349d7670632fab7ef014b8a2e77c0c059bb3039" Sep 06 06:24:06.623430 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:06.623404 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:24:06.623544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:06.623469 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:24:06.623544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:06.623500 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:24:07.629277 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:07.629247 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" event={"ID":"34c9849b-a620-484d-a4e7-627b706e2b01","Type":"ContainerStarted","Data":"5671c9f7114b26e584731ff634c0afd22d84679c97baae9e32cd4890b0a679dd"} Sep 06 06:24:08.429391 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:08.429356 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:24:08.429596 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:08.429574 2578 patch_prober.go:28] interesting pod/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 06 06:24:08.429644 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:08.429623 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" podUID="34c9849b-a620-484d-a4e7-627b706e2b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 06 06:24:09.429735 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:09.429696 2578 patch_prober.go:28] interesting pod/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 06 06:24:09.430187 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:09.429757 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" podUID="34c9849b-a620-484d-a4e7-627b706e2b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 06 06:24:10.430052 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:10.430018 2578 patch_prober.go:28] interesting pod/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 06 06:24:10.430457 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:10.430075 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" podUID="34c9849b-a620-484d-a4e7-627b706e2b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 06 06:24:11.429573 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:11.429537 2578 patch_prober.go:28] interesting pod/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 06 06:24:11.429759 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:11.429594 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" podUID="34c9849b-a620-484d-a4e7-627b706e2b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 06 06:24:12.429387 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:12.429342 2578 patch_prober.go:28] interesting pod/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 06 06:24:12.429859 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:12.429398 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" podUID="34c9849b-a620-484d-a4e7-627b706e2b01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 06 06:24:13.433419 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:13.433386 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:24:13.433859 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:13.433702 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:24:13.434437 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:13.434419 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:24:17.301053 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.301011 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k"] Sep 06 06:24:17.304923 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.304876 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.307243 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.307208 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a52b50-openshift-default-dockercfg-sfwf8\"" Sep 06 06:24:17.317802 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.317768 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k"] Sep 06 06:24:17.326379 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.326337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-workload-socket\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.326514 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.326412 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-workload-certs\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.326514 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.326467 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-data\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.326514 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.326492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbm9k\" (UniqueName: \"kubernetes.io/projected/62e1f506-f005-49aa-8287-d3930a74f0eb-kube-api-access-nbm9k\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.326621 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.326545 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-token\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.326621 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.326586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-credential-socket\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.326692 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.326644 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-podinfo\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.326732 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.326688 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-envoy\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.326732 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.326710 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62e1f506-f005-49aa-8287-d3930a74f0eb-istiod-ca-cert\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.427412 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.427374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-data\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.427412 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.427413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbm9k\" (UniqueName: \"kubernetes.io/projected/62e1f506-f005-49aa-8287-d3930a74f0eb-kube-api-access-nbm9k\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.427673 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.427457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-token\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.427673 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.427494 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-credential-socket\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.427673 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.427542 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-podinfo\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.427673 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.427582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-envoy\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.427673 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.427609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62e1f506-f005-49aa-8287-d3930a74f0eb-istiod-ca-cert\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.427673 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.427649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-workload-socket\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.428020 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.427701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-workload-certs\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.428020 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.427837 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-data\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.428283 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.428259 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-workload-certs\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.428373 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.428278 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-workload-socket\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.428373 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.428346 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-credential-socket\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.428647 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.428556 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62e1f506-f005-49aa-8287-d3930a74f0eb-istiod-ca-cert\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.430201 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.430169 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-envoy\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.430838 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.430808 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-podinfo\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.436162 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.436134 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-token\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.436284 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.436258 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbm9k\" (UniqueName: \"kubernetes.io/projected/62e1f506-f005-49aa-8287-d3930a74f0eb-kube-api-access-nbm9k\") pod \"e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.617788 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.617686 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:17.901580 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.901551 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k"] Sep 06 06:24:17.903670 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:24:17.903641 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e1f506_f005_49aa_8287_d3930a74f0eb.slice/crio-346df172ea6c920dfa47e436f0f624de56ba22f3bb49db1871e98c3e96633e11 WatchSource:0}: Error finding container 346df172ea6c920dfa47e436f0f624de56ba22f3bb49db1871e98c3e96633e11: Status 404 returned error can't find the container with id 346df172ea6c920dfa47e436f0f624de56ba22f3bb49db1871e98c3e96633e11 Sep 06 06:24:17.905943 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.905890 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:24:17.906054 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.905991 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:24:17.906054 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:17.906035 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:24:18.720663 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:18.720628 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" event={"ID":"62e1f506-f005-49aa-8287-d3930a74f0eb","Type":"ContainerStarted","Data":"40dd2af1eefcbdb8782ad30dc5f19403cff8db4303f1b8fb3d74da0f8d807968"} Sep 06 06:24:18.721173 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:18.720681 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" event={"ID":"62e1f506-f005-49aa-8287-d3930a74f0eb","Type":"ContainerStarted","Data":"346df172ea6c920dfa47e436f0f624de56ba22f3bb49db1871e98c3e96633e11"} Sep 06 06:24:18.741580 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:18.741527 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" podStartSLOduration=1.741509997 podStartE2EDuration="1.741509997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:24:18.739083757 +0000 UTC m=+1824.137414236" watchObservedRunningTime="2026-09-06 06:24:18.741509997 +0000 UTC m=+1824.139840464" Sep 06 06:24:19.618668 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:19.618627 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:19.623947 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:19.623894 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:19.726386 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:19.726357 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:19.727267 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:19.727247 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:21.733607 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:21.733547 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9"] Sep 06 06:24:21.738153 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:21.738117 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:21.740362 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:21.740339 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a52b50-serving-cert\"" Sep 06 06:24:21.747157 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:21.747132 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9"] Sep 06 06:24:21.773007 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:21.772983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bwwh\" (UniqueName: \"kubernetes.io/projected/0f86112f-d3bd-4467-817b-6b505f631ddc-kube-api-access-4bwwh\") pod \"maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:21.773137 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:21.773044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f86112f-d3bd-4467-817b-6b505f631ddc-metrics-certs\") pod \"maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:21.773137 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:21.773110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f86112f-d3bd-4467-817b-6b505f631ddc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:21.773227 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:21.773190 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f86112f-d3bd-4467-817b-6b505f631ddc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:21.874174 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:21.874137 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f86112f-d3bd-4467-817b-6b505f631ddc-metrics-certs\") pod \"maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:21.874355 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:21.874229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f86112f-d3bd-4467-817b-6b505f631ddc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:21.874355 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:21.874321 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f86112f-d3bd-4467-817b-6b505f631ddc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:21.874437 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:21.874363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bwwh\" (UniqueName: \"kubernetes.io/projected/0f86112f-d3bd-4467-817b-6b505f631ddc-kube-api-access-4bwwh\") pod \"maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:21.875160 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:24:21.875124 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-serving-cert: secret "maas-api-e2e-ipp-b-a52b50-serving-cert" not found Sep 06 06:24:21.875301 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:24:21.875235 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f86112f-d3bd-4467-817b-6b505f631ddc-maas-api-tls podName:0f86112f-d3bd-4467-817b-6b505f631ddc nodeName:}" failed. No retries permitted until 2026-09-06 06:24:22.375206931 +0000 UTC m=+1827.773537383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f86112f-d3bd-4467-817b-6b505f631ddc-maas-api-tls") pod "maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" (UID: "0f86112f-d3bd-4467-817b-6b505f631ddc") : secret "maas-api-e2e-ipp-b-a52b50-serving-cert" not found Sep 06 06:24:21.875708 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:21.875682 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f86112f-d3bd-4467-817b-6b505f631ddc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:21.882947 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:21.877596 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f86112f-d3bd-4467-817b-6b505f631ddc-metrics-certs\") pod \"maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:21.890476 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:21.890447 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bwwh\" (UniqueName: \"kubernetes.io/projected/0f86112f-d3bd-4467-817b-6b505f631ddc-kube-api-access-4bwwh\") pod \"maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:22.379349 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:22.379310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f86112f-d3bd-4467-817b-6b505f631ddc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:22.382122 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:22.382099 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f86112f-d3bd-4467-817b-6b505f631ddc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:22.653550 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:22.653459 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:22.931938 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:22.931914 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9"] Sep 06 06:24:22.933644 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:24:22.933613 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f86112f_d3bd_4467_817b_6b505f631ddc.slice/crio-867b2da5af2c28242328cee89834f745e4ad22cff744acff7bd634524b1ec5aa WatchSource:0}: Error finding container 867b2da5af2c28242328cee89834f745e4ad22cff744acff7bd634524b1ec5aa: Status 404 returned error can't find the container with id 867b2da5af2c28242328cee89834f745e4ad22cff744acff7bd634524b1ec5aa Sep 06 06:24:23.751183 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:23.751144 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" event={"ID":"0f86112f-d3bd-4467-817b-6b505f631ddc","Type":"ContainerStarted","Data":"c2fb9db033680c4263817b7289689576e6d41ae583cfde467bf974f53c8be07d"} Sep 06 06:24:23.751374 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:23.751208 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" event={"ID":"0f86112f-d3bd-4467-817b-6b505f631ddc","Type":"ContainerStarted","Data":"867b2da5af2c28242328cee89834f745e4ad22cff744acff7bd634524b1ec5aa"} Sep 06 06:24:23.751374 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:23.751235 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:23.766631 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:23.766585 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" podStartSLOduration=2.481310955 podStartE2EDuration="2.766569618s" totalImagesPullingTime="285.258663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:24:23.766248415 +0000 UTC m=+1829.164579107" watchObservedRunningTime="2026-09-06 06:24:23.766569618 +0000 UTC m=+1829.164900130" Sep 06 06:24:28.189534 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:28.189500 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:24:29.772359 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:29.772321 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:24:38.837957 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:38.837923 2578 generic.go:356] "Generic (PLEG): container finished" podID="62e1f506-f005-49aa-8287-d3930a74f0eb" containerID="40dd2af1eefcbdb8782ad30dc5f19403cff8db4303f1b8fb3d74da0f8d807968" exitCode=0 Sep 06 06:24:38.838377 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:38.838006 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" event={"ID":"62e1f506-f005-49aa-8287-d3930a74f0eb","Type":"ContainerDied","Data":"40dd2af1eefcbdb8782ad30dc5f19403cff8db4303f1b8fb3d74da0f8d807968"} Sep 06 06:24:38.838460 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:38.838446 2578 scope.go:117] "RemoveContainer" containerID="40dd2af1eefcbdb8782ad30dc5f19403cff8db4303f1b8fb3d74da0f8d807968" Sep 06 06:24:38.839549 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:38.839513 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:24:38.839665 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:38.839598 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:24:38.839665 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:38.839648 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:24:39.847128 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:39.847096 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" event={"ID":"62e1f506-f005-49aa-8287-d3930a74f0eb","Type":"ContainerStarted","Data":"1e2b06e3678853b507acaae15f05e69fbd5ecf11d8f7a290cffab65d041b3e70"} Sep 06 06:24:40.618102 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:40.618058 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:40.618370 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:40.618345 2578 patch_prober.go:28] interesting pod/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 06 06:24:40.618464 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:40.618419 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" podUID="62e1f506-f005-49aa-8287-d3930a74f0eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 06 06:24:41.619006 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:41.618971 2578 patch_prober.go:28] interesting pod/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 06 06:24:41.619415 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:41.619037 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" podUID="62e1f506-f005-49aa-8287-d3930a74f0eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 06 06:24:42.618633 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:42.618597 2578 patch_prober.go:28] interesting pod/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 06 06:24:42.618818 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:42.618655 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" podUID="62e1f506-f005-49aa-8287-d3930a74f0eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 06 06:24:43.618367 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:43.618331 2578 patch_prober.go:28] interesting pod/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 06 06:24:43.618755 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:43.618405 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" podUID="62e1f506-f005-49aa-8287-d3930a74f0eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 06 06:24:44.618940 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:44.618885 2578 patch_prober.go:28] interesting pod/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 06 06:24:44.619325 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:44.618963 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" podUID="62e1f506-f005-49aa-8287-d3930a74f0eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 06 06:24:45.622730 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:45.622701 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:45.623196 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:45.623054 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:45.623886 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:45.623867 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:24:55.605509 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:24:55.605477 2578 scope.go:117] "RemoveContainer" containerID="93b0c36699a422e4436a3b445b4cbf2780d6fb4953ee5de4935c474ab969c276" Sep 06 06:25:01.888361 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:01.888320 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:25:16.381650 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:16.381574 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:25:23.895473 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:23.895438 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm"] Sep 06 06:25:23.899425 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:23.899404 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:23.901382 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:23.901361 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-3cf67d-openshift-default-dockercfg-lfr6d\"" Sep 06 06:25:23.910095 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:23.910064 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm"] Sep 06 06:25:24.006004 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.005324 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk4x7\" (UniqueName: \"kubernetes.io/projected/53718190-5721-424d-96f8-c7ad42fd349f-kube-api-access-zk4x7\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.006004 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.005389 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53718190-5721-424d-96f8-c7ad42fd349f-istio-podinfo\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.006004 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.005481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-istio-data\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.006004 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.005555 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53718190-5721-424d-96f8-c7ad42fd349f-istiod-ca-cert\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.006004 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.005589 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-credential-socket\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.006004 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.005622 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53718190-5721-424d-96f8-c7ad42fd349f-istio-token\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.006004 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.005664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-istio-envoy\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.006004 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.005729 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-workload-socket\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.006004 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.005768 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-workload-certs\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.106806 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.106771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-workload-socket\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.106806 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.106808 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-workload-certs\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.107099 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.106834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk4x7\" (UniqueName: \"kubernetes.io/projected/53718190-5721-424d-96f8-c7ad42fd349f-kube-api-access-zk4x7\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.107099 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.106865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53718190-5721-424d-96f8-c7ad42fd349f-istio-podinfo\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.107099 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.106941 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-istio-data\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.107099 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.106990 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53718190-5721-424d-96f8-c7ad42fd349f-istiod-ca-cert\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.107099 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.107036 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-credential-socket\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.107099 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.107071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53718190-5721-424d-96f8-c7ad42fd349f-istio-token\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.107442 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.107226 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-workload-socket\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.107442 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.107269 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-workload-certs\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.107442 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.107282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-istio-envoy\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.107442 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.107339 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-istio-data\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.107442 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.107393 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-credential-socket\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.107654 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.107636 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53718190-5721-424d-96f8-c7ad42fd349f-istiod-ca-cert\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.109503 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.109479 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53718190-5721-424d-96f8-c7ad42fd349f-istio-podinfo\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.109620 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.109478 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-istio-envoy\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.114604 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.114580 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53718190-5721-424d-96f8-c7ad42fd349f-istio-token\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.114845 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.114828 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk4x7\" (UniqueName: \"kubernetes.io/projected/53718190-5721-424d-96f8-c7ad42fd349f-kube-api-access-zk4x7\") pod \"e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.210737 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.210646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:24.487663 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.487637 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm"] Sep 06 06:25:24.488967 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:25:24.488938 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53718190_5721_424d_96f8_c7ad42fd349f.slice/crio-cfa6c2687ea4ad65b9960b78f2d20a058f40bb2bc61d2a469fed335757e91067 WatchSource:0}: Error finding container cfa6c2687ea4ad65b9960b78f2d20a058f40bb2bc61d2a469fed335757e91067: Status 404 returned error can't find the container with id cfa6c2687ea4ad65b9960b78f2d20a058f40bb2bc61d2a469fed335757e91067 Sep 06 06:25:24.491362 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.491331 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:25:24.491441 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.491420 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:25:24.491479 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:24.491448 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:25:25.104037 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:25.103999 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" event={"ID":"53718190-5721-424d-96f8-c7ad42fd349f","Type":"ContainerStarted","Data":"47d72f7bdd9cc8ac30ea22b20b7aabcc9fdac45af8330941c9b34f00987f5bd2"} Sep 06 06:25:25.104422 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:25.104058 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" event={"ID":"53718190-5721-424d-96f8-c7ad42fd349f","Type":"ContainerStarted","Data":"cfa6c2687ea4ad65b9960b78f2d20a058f40bb2bc61d2a469fed335757e91067"} Sep 06 06:25:25.130665 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:25.130610 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" podStartSLOduration=2.130594763 podStartE2EDuration="2.130594763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:25:25.1294248 +0000 UTC m=+1890.527755288" watchObservedRunningTime="2026-09-06 06:25:25.130594763 +0000 UTC m=+1890.528925295" Sep 06 06:25:25.211619 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:25.211581 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:25.216821 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:25.216798 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:26.109630 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:26.109602 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:26.110680 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:26.110662 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:28.130355 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.130321 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk"] Sep 06 06:25:28.133823 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.133801 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:28.135987 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.135965 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-3cf67d-serving-cert\"" Sep 06 06:25:28.143004 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.142979 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk"] Sep 06 06:25:28.150680 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.150652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f86b60-42e6-44d9-b627-989b306e5b97-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:28.150793 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.150688 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tntbq\" (UniqueName: \"kubernetes.io/projected/04f86b60-42e6-44d9-b627-989b306e5b97-kube-api-access-tntbq\") pod \"maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:28.150860 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.150830 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04f86b60-42e6-44d9-b627-989b306e5b97-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:28.150964 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.150932 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04f86b60-42e6-44d9-b627-989b306e5b97-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:28.251596 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.251566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f86b60-42e6-44d9-b627-989b306e5b97-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:28.251767 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.251602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tntbq\" (UniqueName: \"kubernetes.io/projected/04f86b60-42e6-44d9-b627-989b306e5b97-kube-api-access-tntbq\") pod \"maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:28.251767 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.251701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04f86b60-42e6-44d9-b627-989b306e5b97-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:28.251767 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.251762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04f86b60-42e6-44d9-b627-989b306e5b97-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:28.252250 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.252232 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f86b60-42e6-44d9-b627-989b306e5b97-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:28.254317 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.254295 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04f86b60-42e6-44d9-b627-989b306e5b97-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:28.254428 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.254348 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04f86b60-42e6-44d9-b627-989b306e5b97-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:28.260182 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.260162 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tntbq\" (UniqueName: \"kubernetes.io/projected/04f86b60-42e6-44d9-b627-989b306e5b97-kube-api-access-tntbq\") pod \"maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:28.448048 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.447944 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:28.722227 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:28.722197 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk"] Sep 06 06:25:28.723755 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:25:28.723727 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f86b60_42e6_44d9_b627_989b306e5b97.slice/crio-4a8566cc76c642d83164d8b44b0a6eaecd77cd380fe6be2003351971fcd6a2af WatchSource:0}: Error finding container 4a8566cc76c642d83164d8b44b0a6eaecd77cd380fe6be2003351971fcd6a2af: Status 404 returned error can't find the container with id 4a8566cc76c642d83164d8b44b0a6eaecd77cd380fe6be2003351971fcd6a2af Sep 06 06:25:29.128494 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:29.128469 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" event={"ID":"04f86b60-42e6-44d9-b627-989b306e5b97","Type":"ContainerStarted","Data":"4a8566cc76c642d83164d8b44b0a6eaecd77cd380fe6be2003351971fcd6a2af"} Sep 06 06:25:30.135575 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:30.135541 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" event={"ID":"04f86b60-42e6-44d9-b627-989b306e5b97","Type":"ContainerStarted","Data":"a1cdc3f6ae54be86e2f2faa5205751bb76037519788c08560f094a77ae1ea527"} Sep 06 06:25:30.135995 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:30.135650 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:30.153735 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:30.153671 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" podStartSLOduration=1.75566321 podStartE2EDuration="2.153653769s" totalImagesPullingTime="397.990559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:25:30.151419739 +0000 UTC m=+1895.549750269" watchObservedRunningTime="2026-09-06 06:25:30.153653769 +0000 UTC m=+1895.551984236" Sep 06 06:25:36.147058 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:36.147014 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:36.479801 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:36.479720 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:25:48.240709 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:48.240667 2578 generic.go:356] "Generic (PLEG): container finished" podID="53718190-5721-424d-96f8-c7ad42fd349f" containerID="47d72f7bdd9cc8ac30ea22b20b7aabcc9fdac45af8330941c9b34f00987f5bd2" exitCode=0 Sep 06 06:25:48.241146 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:48.240738 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" event={"ID":"53718190-5721-424d-96f8-c7ad42fd349f","Type":"ContainerDied","Data":"47d72f7bdd9cc8ac30ea22b20b7aabcc9fdac45af8330941c9b34f00987f5bd2"} Sep 06 06:25:48.241195 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:48.241166 2578 scope.go:117] "RemoveContainer" containerID="47d72f7bdd9cc8ac30ea22b20b7aabcc9fdac45af8330941c9b34f00987f5bd2" Sep 06 06:25:48.242317 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:48.242289 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:25:48.242375 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:48.242353 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:25:48.242414 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:48.242385 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 06 06:25:49.248459 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:49.248427 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" event={"ID":"53718190-5721-424d-96f8-c7ad42fd349f","Type":"ContainerStarted","Data":"523b23b0a87a05352a96cb4c2e2cd01f027c81099c67f7d20d1e1936ee7492de"} Sep 06 06:25:50.211536 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:50.211498 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:50.211813 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:50.211793 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Sep 06 06:25:50.211861 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:50.211835 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" podUID="53718190-5721-424d-96f8-c7ad42fd349f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Sep 06 06:25:51.211523 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:51.211480 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Sep 06 06:25:51.211965 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:51.211543 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" podUID="53718190-5721-424d-96f8-c7ad42fd349f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Sep 06 06:25:52.211193 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:52.211158 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Sep 06 06:25:52.211373 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:52.211219 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" podUID="53718190-5721-424d-96f8-c7ad42fd349f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Sep 06 06:25:53.211307 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:53.211271 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Sep 06 06:25:53.211723 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:53.211327 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" podUID="53718190-5721-424d-96f8-c7ad42fd349f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Sep 06 06:25:54.211605 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:54.211568 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Sep 06 06:25:54.212048 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:54.211625 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" podUID="53718190-5721-424d-96f8-c7ad42fd349f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Sep 06 06:25:55.215727 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:55.215694 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:55.216123 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:55.215940 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:55.216812 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:55.216795 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:25:58.629938 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:58.629861 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk"] Sep 06 06:25:58.630452 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:58.630210 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" podUID="04f86b60-42e6-44d9-b627-989b306e5b97" containerName="maas-api" containerID="cri-o://a1cdc3f6ae54be86e2f2faa5205751bb76037519788c08560f094a77ae1ea527" gracePeriod=30 Sep 06 06:25:58.883848 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:58.883791 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:58.973394 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:58.973363 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04f86b60-42e6-44d9-b627-989b306e5b97-maas-api-tls\") pod \"04f86b60-42e6-44d9-b627-989b306e5b97\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " Sep 06 06:25:58.973567 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:58.973534 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tntbq\" (UniqueName: \"kubernetes.io/projected/04f86b60-42e6-44d9-b627-989b306e5b97-kube-api-access-tntbq\") pod \"04f86b60-42e6-44d9-b627-989b306e5b97\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " Sep 06 06:25:58.973621 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:58.973573 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04f86b60-42e6-44d9-b627-989b306e5b97-metrics-certs\") pod \"04f86b60-42e6-44d9-b627-989b306e5b97\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " Sep 06 06:25:58.973621 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:58.973592 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f86b60-42e6-44d9-b627-989b306e5b97-openshift-service-ca\") pod \"04f86b60-42e6-44d9-b627-989b306e5b97\" (UID: \"04f86b60-42e6-44d9-b627-989b306e5b97\") " Sep 06 06:25:58.974175 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:58.974065 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f86b60-42e6-44d9-b627-989b306e5b97-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "04f86b60-42e6-44d9-b627-989b306e5b97" (UID: "04f86b60-42e6-44d9-b627-989b306e5b97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:25:58.975757 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:58.975734 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f86b60-42e6-44d9-b627-989b306e5b97-kube-api-access-tntbq" (OuterVolumeSpecName: "kube-api-access-tntbq") pod "04f86b60-42e6-44d9-b627-989b306e5b97" (UID: "04f86b60-42e6-44d9-b627-989b306e5b97"). InnerVolumeSpecName "kube-api-access-tntbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:25:58.975825 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:58.975758 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04f86b60-42e6-44d9-b627-989b306e5b97-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "04f86b60-42e6-44d9-b627-989b306e5b97" (UID: "04f86b60-42e6-44d9-b627-989b306e5b97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:25:58.975825 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:58.975810 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04f86b60-42e6-44d9-b627-989b306e5b97-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "04f86b60-42e6-44d9-b627-989b306e5b97" (UID: "04f86b60-42e6-44d9-b627-989b306e5b97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:25:59.074429 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:59.074398 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04f86b60-42e6-44d9-b627-989b306e5b97-maas-api-tls\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:25:59.074429 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:59.074427 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tntbq\" (UniqueName: \"kubernetes.io/projected/04f86b60-42e6-44d9-b627-989b306e5b97-kube-api-access-tntbq\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:25:59.074429 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:59.074438 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04f86b60-42e6-44d9-b627-989b306e5b97-metrics-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:25:59.074670 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:59.074447 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f86b60-42e6-44d9-b627-989b306e5b97-openshift-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:25:59.313035 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:59.313005 2578 generic.go:356] "Generic (PLEG): container finished" podID="04f86b60-42e6-44d9-b627-989b306e5b97" containerID="a1cdc3f6ae54be86e2f2faa5205751bb76037519788c08560f094a77ae1ea527" exitCode=0 Sep 06 06:25:59.313198 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:59.313051 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" Sep 06 06:25:59.313198 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:59.313097 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" event={"ID":"04f86b60-42e6-44d9-b627-989b306e5b97","Type":"ContainerDied","Data":"a1cdc3f6ae54be86e2f2faa5205751bb76037519788c08560f094a77ae1ea527"} Sep 06 06:25:59.313198 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:59.313152 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk" event={"ID":"04f86b60-42e6-44d9-b627-989b306e5b97","Type":"ContainerDied","Data":"4a8566cc76c642d83164d8b44b0a6eaecd77cd380fe6be2003351971fcd6a2af"} Sep 06 06:25:59.313198 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:59.313179 2578 scope.go:117] "RemoveContainer" containerID="a1cdc3f6ae54be86e2f2faa5205751bb76037519788c08560f094a77ae1ea527" Sep 06 06:25:59.322814 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:59.322796 2578 scope.go:117] "RemoveContainer" containerID="a1cdc3f6ae54be86e2f2faa5205751bb76037519788c08560f094a77ae1ea527" Sep 06 06:25:59.323137 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:25:59.323117 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1cdc3f6ae54be86e2f2faa5205751bb76037519788c08560f094a77ae1ea527\": container with ID starting with a1cdc3f6ae54be86e2f2faa5205751bb76037519788c08560f094a77ae1ea527 not found: ID does not exist" containerID="a1cdc3f6ae54be86e2f2faa5205751bb76037519788c08560f094a77ae1ea527" Sep 06 06:25:59.323200 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:59.323146 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1cdc3f6ae54be86e2f2faa5205751bb76037519788c08560f094a77ae1ea527"} err="failed to get container status \"a1cdc3f6ae54be86e2f2faa5205751bb76037519788c08560f094a77ae1ea527\": rpc error: code = NotFound desc = could not find container \"a1cdc3f6ae54be86e2f2faa5205751bb76037519788c08560f094a77ae1ea527\": container with ID starting with a1cdc3f6ae54be86e2f2faa5205751bb76037519788c08560f094a77ae1ea527 not found: ID does not exist" Sep 06 06:25:59.329105 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:59.329071 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk"] Sep 06 06:25:59.331401 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:25:59.331381 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3cf67d-f84b8754b-t9rkk"] Sep 06 06:26:01.144859 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:01.144823 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04f86b60-42e6-44d9-b627-989b306e5b97" path="/var/lib/kubelet/pods/04f86b60-42e6-44d9-b627-989b306e5b97/volumes" Sep 06 06:26:04.173073 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:04.173038 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:26:05.258869 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:05.258833 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm"] Sep 06 06:26:05.259273 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:05.259051 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" podUID="53718190-5721-424d-96f8-c7ad42fd349f" containerName="istio-proxy" containerID="cri-o://523b23b0a87a05352a96cb4c2e2cd01f027c81099c67f7d20d1e1936ee7492de" gracePeriod=30 Sep 06 06:26:10.218839 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.218791 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 06 06:26:10.219255 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.218869 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" podUID="53718190-5721-424d-96f8-c7ad42fd349f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 06 06:26:10.392137 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.392105 2578 generic.go:356] "Generic (PLEG): container finished" podID="53718190-5721-424d-96f8-c7ad42fd349f" containerID="523b23b0a87a05352a96cb4c2e2cd01f027c81099c67f7d20d1e1936ee7492de" exitCode=0 Sep 06 06:26:10.392328 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.392176 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" event={"ID":"53718190-5721-424d-96f8-c7ad42fd349f","Type":"ContainerDied","Data":"523b23b0a87a05352a96cb4c2e2cd01f027c81099c67f7d20d1e1936ee7492de"} Sep 06 06:26:10.392328 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.392236 2578 scope.go:117] "RemoveContainer" containerID="47d72f7bdd9cc8ac30ea22b20b7aabcc9fdac45af8330941c9b34f00987f5bd2" Sep 06 06:26:10.528042 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.528016 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:26:10.587223 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.587195 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-istio-envoy\") pod \"53718190-5721-424d-96f8-c7ad42fd349f\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " Sep 06 06:26:10.587389 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.587237 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53718190-5721-424d-96f8-c7ad42fd349f-istio-token\") pod \"53718190-5721-424d-96f8-c7ad42fd349f\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " Sep 06 06:26:10.587389 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.587268 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-workload-socket\") pod \"53718190-5721-424d-96f8-c7ad42fd349f\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " Sep 06 06:26:10.587389 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.587298 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-credential-socket\") pod \"53718190-5721-424d-96f8-c7ad42fd349f\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " Sep 06 06:26:10.587389 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.587330 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk4x7\" (UniqueName: \"kubernetes.io/projected/53718190-5721-424d-96f8-c7ad42fd349f-kube-api-access-zk4x7\") pod \"53718190-5721-424d-96f8-c7ad42fd349f\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " Sep 06 06:26:10.587604 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.587396 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53718190-5721-424d-96f8-c7ad42fd349f-istiod-ca-cert\") pod \"53718190-5721-424d-96f8-c7ad42fd349f\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " Sep 06 06:26:10.587604 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.587440 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53718190-5721-424d-96f8-c7ad42fd349f-istio-podinfo\") pod \"53718190-5721-424d-96f8-c7ad42fd349f\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " Sep 06 06:26:10.587604 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.587477 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-istio-data\") pod \"53718190-5721-424d-96f8-c7ad42fd349f\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " Sep 06 06:26:10.587604 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.587506 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-workload-certs\") pod \"53718190-5721-424d-96f8-c7ad42fd349f\" (UID: \"53718190-5721-424d-96f8-c7ad42fd349f\") " Sep 06 06:26:10.587827 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.587619 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "53718190-5721-424d-96f8-c7ad42fd349f" (UID: "53718190-5721-424d-96f8-c7ad42fd349f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:26:10.587827 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.587605 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "53718190-5721-424d-96f8-c7ad42fd349f" (UID: "53718190-5721-424d-96f8-c7ad42fd349f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:26:10.587958 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.587832 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-istio-data" (OuterVolumeSpecName: "istio-data") pod "53718190-5721-424d-96f8-c7ad42fd349f" (UID: "53718190-5721-424d-96f8-c7ad42fd349f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:26:10.587958 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.587849 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "53718190-5721-424d-96f8-c7ad42fd349f" (UID: "53718190-5721-424d-96f8-c7ad42fd349f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:26:10.588240 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.588025 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53718190-5721-424d-96f8-c7ad42fd349f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "53718190-5721-424d-96f8-c7ad42fd349f" (UID: "53718190-5721-424d-96f8-c7ad42fd349f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:26:10.588240 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.588142 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:26:10.588240 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.588163 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:26:10.588240 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.588178 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:26:10.588240 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.588191 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:26:10.589591 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.589560 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53718190-5721-424d-96f8-c7ad42fd349f-istio-token" (OuterVolumeSpecName: "istio-token") pod "53718190-5721-424d-96f8-c7ad42fd349f" (UID: "53718190-5721-424d-96f8-c7ad42fd349f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:26:10.589591 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.589574 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "53718190-5721-424d-96f8-c7ad42fd349f" (UID: "53718190-5721-424d-96f8-c7ad42fd349f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:26:10.589755 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.589663 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53718190-5721-424d-96f8-c7ad42fd349f-kube-api-access-zk4x7" (OuterVolumeSpecName: "kube-api-access-zk4x7") pod "53718190-5721-424d-96f8-c7ad42fd349f" (UID: "53718190-5721-424d-96f8-c7ad42fd349f"). InnerVolumeSpecName "kube-api-access-zk4x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:26:10.589819 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.589773 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53718190-5721-424d-96f8-c7ad42fd349f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "53718190-5721-424d-96f8-c7ad42fd349f" (UID: "53718190-5721-424d-96f8-c7ad42fd349f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:26:10.688871 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.688837 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53718190-5721-424d-96f8-c7ad42fd349f-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:26:10.688871 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.688865 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53718190-5721-424d-96f8-c7ad42fd349f-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:26:10.688871 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.688877 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk4x7\" (UniqueName: \"kubernetes.io/projected/53718190-5721-424d-96f8-c7ad42fd349f-kube-api-access-zk4x7\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:26:10.689128 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.688886 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53718190-5721-424d-96f8-c7ad42fd349f-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:26:10.689128 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:10.688895 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53718190-5721-424d-96f8-c7ad42fd349f-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:26:11.399182 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:11.399154 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" Sep 06 06:26:11.399182 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:11.399171 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm" event={"ID":"53718190-5721-424d-96f8-c7ad42fd349f","Type":"ContainerDied","Data":"cfa6c2687ea4ad65b9960b78f2d20a058f40bb2bc61d2a469fed335757e91067"} Sep 06 06:26:11.399634 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:11.399223 2578 scope.go:117] "RemoveContainer" containerID="523b23b0a87a05352a96cb4c2e2cd01f027c81099c67f7d20d1e1936ee7492de" Sep 06 06:26:11.415380 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:11.415356 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm"] Sep 06 06:26:11.419795 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:11.419773 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3cf67d-openshift-default-7d6d8db66b-fmtrm"] Sep 06 06:26:13.144978 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:13.144943 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53718190-5721-424d-96f8-c7ad42fd349f" path="/var/lib/kubelet/pods/53718190-5721-424d-96f8-c7ad42fd349f/volumes" Sep 06 06:26:24.218974 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.218936 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn"] Sep 06 06:26:24.219549 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.219213 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" podUID="c2bafea7-a0de-4229-b009-cee4614b0afb" containerName="maas-api" containerID="cri-o://7467c641ad4ea318b4dfb9f2ad33ea3b80717569d6108cd4951ed79eb0832504" gracePeriod=30 Sep 06 06:26:24.472510 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.472443 2578 generic.go:356] "Generic (PLEG): container finished" podID="c2bafea7-a0de-4229-b009-cee4614b0afb" containerID="7467c641ad4ea318b4dfb9f2ad33ea3b80717569d6108cd4951ed79eb0832504" exitCode=0 Sep 06 06:26:24.472636 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.472512 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" event={"ID":"c2bafea7-a0de-4229-b009-cee4614b0afb","Type":"ContainerDied","Data":"7467c641ad4ea318b4dfb9f2ad33ea3b80717569d6108cd4951ed79eb0832504"} Sep 06 06:26:24.472636 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.472561 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" event={"ID":"c2bafea7-a0de-4229-b009-cee4614b0afb","Type":"ContainerDied","Data":"191d02190aa286bdd2645f6bd0b97b90c997351d3456ed8008564aa319147350"} Sep 06 06:26:24.472636 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.472577 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="191d02190aa286bdd2645f6bd0b97b90c997351d3456ed8008564aa319147350" Sep 06 06:26:24.473956 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.473940 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:26:24.529132 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.529100 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bafea7-a0de-4229-b009-cee4614b0afb-openshift-service-ca\") pod \"c2bafea7-a0de-4229-b009-cee4614b0afb\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " Sep 06 06:26:24.529318 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.529140 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk9jj\" (UniqueName: \"kubernetes.io/projected/c2bafea7-a0de-4229-b009-cee4614b0afb-kube-api-access-wk9jj\") pod \"c2bafea7-a0de-4229-b009-cee4614b0afb\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " Sep 06 06:26:24.529318 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.529213 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-maas-api-tls\") pod \"c2bafea7-a0de-4229-b009-cee4614b0afb\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " Sep 06 06:26:24.529318 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.529233 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-metrics-certs\") pod \"c2bafea7-a0de-4229-b009-cee4614b0afb\" (UID: \"c2bafea7-a0de-4229-b009-cee4614b0afb\") " Sep 06 06:26:24.529546 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.529520 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2bafea7-a0de-4229-b009-cee4614b0afb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2bafea7-a0de-4229-b009-cee4614b0afb" (UID: "c2bafea7-a0de-4229-b009-cee4614b0afb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:26:24.531599 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.531571 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2bafea7-a0de-4229-b009-cee4614b0afb-kube-api-access-wk9jj" (OuterVolumeSpecName: "kube-api-access-wk9jj") pod "c2bafea7-a0de-4229-b009-cee4614b0afb" (UID: "c2bafea7-a0de-4229-b009-cee4614b0afb"). InnerVolumeSpecName "kube-api-access-wk9jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:26:24.531599 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.531583 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c2bafea7-a0de-4229-b009-cee4614b0afb" (UID: "c2bafea7-a0de-4229-b009-cee4614b0afb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:26:24.531755 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.531629 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2bafea7-a0de-4229-b009-cee4614b0afb" (UID: "c2bafea7-a0de-4229-b009-cee4614b0afb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:26:24.630734 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.630689 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-maas-api-tls\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:26:24.630734 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.630736 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2bafea7-a0de-4229-b009-cee4614b0afb-metrics-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:26:24.631071 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.630751 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bafea7-a0de-4229-b009-cee4614b0afb-openshift-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:26:24.631071 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.630769 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk9jj\" (UniqueName: \"kubernetes.io/projected/c2bafea7-a0de-4229-b009-cee4614b0afb-kube-api-access-wk9jj\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:26:24.980269 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:24.980234 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:26:25.478839 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:25.478804 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn" Sep 06 06:26:25.494951 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:25.494925 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn"] Sep 06 06:26:25.498336 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:25.498315 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a36847-76f6b6d867-pcxjn"] Sep 06 06:26:27.144927 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:27.144879 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2bafea7-a0de-4229-b009-cee4614b0afb" path="/var/lib/kubelet/pods/c2bafea7-a0de-4229-b009-cee4614b0afb/volumes" Sep 06 06:26:34.988588 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:34.988549 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:26:55.636013 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:26:55.635938 2578 scope.go:117] "RemoveContainer" containerID="d58e901fb8926455c31e02712ad460346a922f436adfcdd4b80f28cf21d82d9d" Sep 06 06:27:26.662266 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:26.662228 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6"] Sep 06 06:27:26.662869 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:26.662538 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" podUID="34c9849b-a620-484d-a4e7-627b706e2b01" containerName="istio-proxy" containerID="cri-o://5671c9f7114b26e584731ff634c0afd22d84679c97baae9e32cd4890b0a679dd" gracePeriod=30 Sep 06 06:27:28.436776 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:28.436734 2578 patch_prober.go:28] interesting pod/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 06 06:27:28.437169 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:28.436801 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" podUID="34c9849b-a620-484d-a4e7-627b706e2b01" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 06 06:27:31.828964 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.828897 2578 generic.go:356] "Generic (PLEG): container finished" podID="34c9849b-a620-484d-a4e7-627b706e2b01" containerID="5671c9f7114b26e584731ff634c0afd22d84679c97baae9e32cd4890b0a679dd" exitCode=0 Sep 06 06:27:31.828964 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.828942 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" event={"ID":"34c9849b-a620-484d-a4e7-627b706e2b01","Type":"ContainerDied","Data":"5671c9f7114b26e584731ff634c0afd22d84679c97baae9e32cd4890b0a679dd"} Sep 06 06:27:31.829346 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.829041 2578 scope.go:117] "RemoveContainer" containerID="588fcde7a52bcda484a350479349d7670632fab7ef014b8a2e77c0c059bb3039" Sep 06 06:27:31.916883 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.916860 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:27:31.995247 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995167 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-istio-envoy\") pod \"34c9849b-a620-484d-a4e7-627b706e2b01\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " Sep 06 06:27:31.995247 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995228 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-credential-socket\") pod \"34c9849b-a620-484d-a4e7-627b706e2b01\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " Sep 06 06:27:31.995478 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995261 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-istio-data\") pod \"34c9849b-a620-484d-a4e7-627b706e2b01\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " Sep 06 06:27:31.995478 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995291 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c9849b-a620-484d-a4e7-627b706e2b01-istio-token\") pod \"34c9849b-a620-484d-a4e7-627b706e2b01\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " Sep 06 06:27:31.995478 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995317 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c9849b-a620-484d-a4e7-627b706e2b01-istiod-ca-cert\") pod \"34c9849b-a620-484d-a4e7-627b706e2b01\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " Sep 06 06:27:31.995478 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995334 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c9849b-a620-484d-a4e7-627b706e2b01-istio-podinfo\") pod \"34c9849b-a620-484d-a4e7-627b706e2b01\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " Sep 06 06:27:31.995478 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995387 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-workload-certs\") pod \"34c9849b-a620-484d-a4e7-627b706e2b01\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " Sep 06 06:27:31.995478 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995434 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k5tw\" (UniqueName: \"kubernetes.io/projected/34c9849b-a620-484d-a4e7-627b706e2b01-kube-api-access-2k5tw\") pod \"34c9849b-a620-484d-a4e7-627b706e2b01\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " Sep 06 06:27:31.995478 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995465 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-workload-socket\") pod \"34c9849b-a620-484d-a4e7-627b706e2b01\" (UID: \"34c9849b-a620-484d-a4e7-627b706e2b01\") " Sep 06 06:27:31.995830 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995572 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "34c9849b-a620-484d-a4e7-627b706e2b01" (UID: "34c9849b-a620-484d-a4e7-627b706e2b01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:27:31.995830 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995586 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-istio-data" (OuterVolumeSpecName: "istio-data") pod "34c9849b-a620-484d-a4e7-627b706e2b01" (UID: "34c9849b-a620-484d-a4e7-627b706e2b01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:27:31.995830 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995655 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "34c9849b-a620-484d-a4e7-627b706e2b01" (UID: "34c9849b-a620-484d-a4e7-627b706e2b01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:27:31.995830 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995759 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "34c9849b-a620-484d-a4e7-627b706e2b01" (UID: "34c9849b-a620-484d-a4e7-627b706e2b01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:27:31.995830 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995706 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c9849b-a620-484d-a4e7-627b706e2b01-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "34c9849b-a620-484d-a4e7-627b706e2b01" (UID: "34c9849b-a620-484d-a4e7-627b706e2b01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:27:31.996132 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995949 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:31.996132 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995970 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:31.996132 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995984 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:31.996132 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.995996 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c9849b-a620-484d-a4e7-627b706e2b01-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:31.996132 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.996006 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:31.997558 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.997535 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c9849b-a620-484d-a4e7-627b706e2b01-istio-token" (OuterVolumeSpecName: "istio-token") pod "34c9849b-a620-484d-a4e7-627b706e2b01" (UID: "34c9849b-a620-484d-a4e7-627b706e2b01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:27:31.997674 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.997660 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34c9849b-a620-484d-a4e7-627b706e2b01-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "34c9849b-a620-484d-a4e7-627b706e2b01" (UID: "34c9849b-a620-484d-a4e7-627b706e2b01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:27:31.997716 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.997659 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "34c9849b-a620-484d-a4e7-627b706e2b01" (UID: "34c9849b-a620-484d-a4e7-627b706e2b01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:27:31.997783 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:31.997767 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c9849b-a620-484d-a4e7-627b706e2b01-kube-api-access-2k5tw" (OuterVolumeSpecName: "kube-api-access-2k5tw") pod "34c9849b-a620-484d-a4e7-627b706e2b01" (UID: "34c9849b-a620-484d-a4e7-627b706e2b01"). InnerVolumeSpecName "kube-api-access-2k5tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:27:32.096575 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:32.096537 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c9849b-a620-484d-a4e7-627b706e2b01-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:32.096575 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:32.096569 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c9849b-a620-484d-a4e7-627b706e2b01-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:32.096575 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:32.096580 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k5tw\" (UniqueName: \"kubernetes.io/projected/34c9849b-a620-484d-a4e7-627b706e2b01-kube-api-access-2k5tw\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:32.096822 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:32.096591 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c9849b-a620-484d-a4e7-627b706e2b01-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:32.836415 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:32.836387 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" Sep 06 06:27:32.836811 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:32.836433 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6" event={"ID":"34c9849b-a620-484d-a4e7-627b706e2b01","Type":"ContainerDied","Data":"4c3a7b583e0a7f96aa28c2339b23a1e9a641f4f281c58caae45aca273d2c2053"} Sep 06 06:27:32.836811 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:32.836493 2578 scope.go:117] "RemoveContainer" containerID="5671c9f7114b26e584731ff634c0afd22d84679c97baae9e32cd4890b0a679dd" Sep 06 06:27:32.857346 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:32.857322 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6"] Sep 06 06:27:32.862208 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:32.862188 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a36847-openshift-default-5b54cb44cb-72gt6"] Sep 06 06:27:33.145640 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:33.145605 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34c9849b-a620-484d-a4e7-627b706e2b01" path="/var/lib/kubelet/pods/34c9849b-a620-484d-a4e7-627b706e2b01/volumes" Sep 06 06:27:37.486817 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.486781 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9"] Sep 06 06:27:37.487304 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.487048 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" podUID="0f86112f-d3bd-4467-817b-6b505f631ddc" containerName="maas-api" containerID="cri-o://c2fb9db033680c4263817b7289689576e6d41ae583cfde467bf974f53c8be07d" gracePeriod=30 Sep 06 06:27:37.734660 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.734637 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:27:37.852786 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.852748 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f86112f-d3bd-4467-817b-6b505f631ddc-maas-api-tls\") pod \"0f86112f-d3bd-4467-817b-6b505f631ddc\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " Sep 06 06:27:37.852997 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.852811 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bwwh\" (UniqueName: \"kubernetes.io/projected/0f86112f-d3bd-4467-817b-6b505f631ddc-kube-api-access-4bwwh\") pod \"0f86112f-d3bd-4467-817b-6b505f631ddc\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " Sep 06 06:27:37.852997 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.852888 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f86112f-d3bd-4467-817b-6b505f631ddc-openshift-service-ca\") pod \"0f86112f-d3bd-4467-817b-6b505f631ddc\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " Sep 06 06:27:37.853150 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.853023 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f86112f-d3bd-4467-817b-6b505f631ddc-metrics-certs\") pod \"0f86112f-d3bd-4467-817b-6b505f631ddc\" (UID: \"0f86112f-d3bd-4467-817b-6b505f631ddc\") " Sep 06 06:27:37.853251 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.853225 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f86112f-d3bd-4467-817b-6b505f631ddc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f86112f-d3bd-4467-817b-6b505f631ddc" (UID: "0f86112f-d3bd-4467-817b-6b505f631ddc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:27:37.853360 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.853334 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f86112f-d3bd-4467-817b-6b505f631ddc-openshift-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:37.855152 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.855127 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f86112f-d3bd-4467-817b-6b505f631ddc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0f86112f-d3bd-4467-817b-6b505f631ddc" (UID: "0f86112f-d3bd-4467-817b-6b505f631ddc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:27:37.855152 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.855140 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f86112f-d3bd-4467-817b-6b505f631ddc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f86112f-d3bd-4467-817b-6b505f631ddc" (UID: "0f86112f-d3bd-4467-817b-6b505f631ddc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:27:37.855303 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.855252 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f86112f-d3bd-4467-817b-6b505f631ddc-kube-api-access-4bwwh" (OuterVolumeSpecName: "kube-api-access-4bwwh") pod "0f86112f-d3bd-4467-817b-6b505f631ddc" (UID: "0f86112f-d3bd-4467-817b-6b505f631ddc"). InnerVolumeSpecName "kube-api-access-4bwwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:27:37.876372 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.876337 2578 generic.go:356] "Generic (PLEG): container finished" podID="0f86112f-d3bd-4467-817b-6b505f631ddc" containerID="c2fb9db033680c4263817b7289689576e6d41ae583cfde467bf974f53c8be07d" exitCode=0 Sep 06 06:27:37.876557 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.876382 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" Sep 06 06:27:37.876557 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.876406 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" event={"ID":"0f86112f-d3bd-4467-817b-6b505f631ddc","Type":"ContainerDied","Data":"c2fb9db033680c4263817b7289689576e6d41ae583cfde467bf974f53c8be07d"} Sep 06 06:27:37.876557 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.876458 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9" event={"ID":"0f86112f-d3bd-4467-817b-6b505f631ddc","Type":"ContainerDied","Data":"867b2da5af2c28242328cee89834f745e4ad22cff744acff7bd634524b1ec5aa"} Sep 06 06:27:37.876557 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.876485 2578 scope.go:117] "RemoveContainer" containerID="c2fb9db033680c4263817b7289689576e6d41ae583cfde467bf974f53c8be07d" Sep 06 06:27:37.890262 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.890147 2578 scope.go:117] "RemoveContainer" containerID="c2fb9db033680c4263817b7289689576e6d41ae583cfde467bf974f53c8be07d" Sep 06 06:27:37.891446 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:27:37.891414 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2fb9db033680c4263817b7289689576e6d41ae583cfde467bf974f53c8be07d\": container with ID starting with c2fb9db033680c4263817b7289689576e6d41ae583cfde467bf974f53c8be07d not found: ID does not exist" containerID="c2fb9db033680c4263817b7289689576e6d41ae583cfde467bf974f53c8be07d" Sep 06 06:27:37.891582 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.891458 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2fb9db033680c4263817b7289689576e6d41ae583cfde467bf974f53c8be07d"} err="failed to get container status \"c2fb9db033680c4263817b7289689576e6d41ae583cfde467bf974f53c8be07d\": rpc error: code = NotFound desc = could not find container \"c2fb9db033680c4263817b7289689576e6d41ae583cfde467bf974f53c8be07d\": container with ID starting with c2fb9db033680c4263817b7289689576e6d41ae583cfde467bf974f53c8be07d not found: ID does not exist" Sep 06 06:27:37.901919 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.901870 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9"] Sep 06 06:27:37.905217 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.905192 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a52b50-5bd7968f58-dgzd9"] Sep 06 06:27:37.954653 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.954622 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f86112f-d3bd-4467-817b-6b505f631ddc-metrics-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:37.954653 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.954647 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f86112f-d3bd-4467-817b-6b505f631ddc-maas-api-tls\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:37.954653 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:37.954657 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bwwh\" (UniqueName: \"kubernetes.io/projected/0f86112f-d3bd-4467-817b-6b505f631ddc-kube-api-access-4bwwh\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:39.145426 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:39.145395 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f86112f-d3bd-4467-817b-6b505f631ddc" path="/var/lib/kubelet/pods/0f86112f-d3bd-4467-817b-6b505f631ddc/volumes" Sep 06 06:27:40.781130 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:40.781090 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:27:44.083871 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.083839 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k"] Sep 06 06:27:44.084285 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.084111 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" podUID="62e1f506-f005-49aa-8287-d3930a74f0eb" containerName="istio-proxy" containerID="cri-o://1e2b06e3678853b507acaae15f05e69fbd5ecf11d8f7a290cffab65d041b3e70" gracePeriod=30 Sep 06 06:27:44.452965 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.452862 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt"] Sep 06 06:27:44.453258 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453244 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04f86b60-42e6-44d9-b627-989b306e5b97" containerName="maas-api" Sep 06 06:27:44.453301 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453261 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="04f86b60-42e6-44d9-b627-989b306e5b97" containerName="maas-api" Sep 06 06:27:44.453301 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453281 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53718190-5721-424d-96f8-c7ad42fd349f" containerName="istio-proxy" Sep 06 06:27:44.453301 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453286 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53718190-5721-424d-96f8-c7ad42fd349f" containerName="istio-proxy" Sep 06 06:27:44.453301 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453294 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f86112f-d3bd-4467-817b-6b505f631ddc" containerName="maas-api" Sep 06 06:27:44.453301 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453299 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f86112f-d3bd-4467-817b-6b505f631ddc" containerName="maas-api" Sep 06 06:27:44.453453 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453310 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34c9849b-a620-484d-a4e7-627b706e2b01" containerName="istio-proxy" Sep 06 06:27:44.453453 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453315 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34c9849b-a620-484d-a4e7-627b706e2b01" containerName="istio-proxy" Sep 06 06:27:44.453453 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453322 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2bafea7-a0de-4229-b009-cee4614b0afb" containerName="maas-api" Sep 06 06:27:44.453453 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453327 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2bafea7-a0de-4229-b009-cee4614b0afb" containerName="maas-api" Sep 06 06:27:44.453453 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453332 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53718190-5721-424d-96f8-c7ad42fd349f" containerName="istio-proxy" Sep 06 06:27:44.453453 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453336 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53718190-5721-424d-96f8-c7ad42fd349f" containerName="istio-proxy" Sep 06 06:27:44.453453 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453344 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34c9849b-a620-484d-a4e7-627b706e2b01" containerName="istio-proxy" Sep 06 06:27:44.453453 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453348 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34c9849b-a620-484d-a4e7-627b706e2b01" containerName="istio-proxy" Sep 06 06:27:44.453453 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453430 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2bafea7-a0de-4229-b009-cee4614b0afb" containerName="maas-api" Sep 06 06:27:44.453453 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453437 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="04f86b60-42e6-44d9-b627-989b306e5b97" containerName="maas-api" Sep 06 06:27:44.453453 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453444 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f86112f-d3bd-4467-817b-6b505f631ddc" containerName="maas-api" Sep 06 06:27:44.453453 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453448 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53718190-5721-424d-96f8-c7ad42fd349f" containerName="istio-proxy" Sep 06 06:27:44.453453 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453453 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53718190-5721-424d-96f8-c7ad42fd349f" containerName="istio-proxy" Sep 06 06:27:44.453832 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453459 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="34c9849b-a620-484d-a4e7-627b706e2b01" containerName="istio-proxy" Sep 06 06:27:44.453832 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.453466 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="34c9849b-a620-484d-a4e7-627b706e2b01" containerName="istio-proxy" Sep 06 06:27:44.458079 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.458062 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt" Sep 06 06:27:44.460262 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.460219 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:27:44.460262 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.460219 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:27:44.463377 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.463354 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt"] Sep 06 06:27:44.618597 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.618550 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5221ee38-495a-4653-beb1-df79c0c6d2fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt\" (UID: \"5221ee38-495a-4653-beb1-df79c0c6d2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt" Sep 06 06:27:44.720154 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.720050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5221ee38-495a-4653-beb1-df79c0c6d2fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt\" (UID: \"5221ee38-495a-4653-beb1-df79c0c6d2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt" Sep 06 06:27:44.720639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.720618 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5221ee38-495a-4653-beb1-df79c0c6d2fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt\" (UID: \"5221ee38-495a-4653-beb1-df79c0c6d2fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt" Sep 06 06:27:44.769828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:44.769795 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt" Sep 06 06:27:45.036916 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:45.036868 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt"] Sep 06 06:27:45.038527 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:27:45.038502 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5221ee38_495a_4653_beb1_df79c0c6d2fd.slice/crio-da60749d2e4ab9d2dbbd1321a5d1361eceefd3a403faad7482b5f98e5e4d5c57 WatchSource:0}: Error finding container da60749d2e4ab9d2dbbd1321a5d1361eceefd3a403faad7482b5f98e5e4d5c57: Status 404 returned error can't find the container with id da60749d2e4ab9d2dbbd1321a5d1361eceefd3a403faad7482b5f98e5e4d5c57 Sep 06 06:27:45.625445 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:45.625407 2578 patch_prober.go:28] interesting pod/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 06 06:27:45.625877 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:45.625464 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" podUID="62e1f506-f005-49aa-8287-d3930a74f0eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 06 06:27:45.921579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:45.921492 2578 generic.go:356] "Generic (PLEG): container finished" podID="5221ee38-495a-4653-beb1-df79c0c6d2fd" containerID="d0ff7108ce3727d3717eb58f0c0f4f7b88990de22016f810adb3ff27f8dbf28a" exitCode=0 Sep 06 06:27:45.921579 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:45.921542 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt" event={"ID":"5221ee38-495a-4653-beb1-df79c0c6d2fd","Type":"ContainerDied","Data":"d0ff7108ce3727d3717eb58f0c0f4f7b88990de22016f810adb3ff27f8dbf28a"} Sep 06 06:27:45.921762 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:45.921592 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt" event={"ID":"5221ee38-495a-4653-beb1-df79c0c6d2fd","Type":"ContainerStarted","Data":"da60749d2e4ab9d2dbbd1321a5d1361eceefd3a403faad7482b5f98e5e4d5c57"} Sep 06 06:27:47.060331 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:47.060300 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt" Sep 06 06:27:47.144885 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:47.144851 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5221ee38-495a-4653-beb1-df79c0c6d2fd-maas-api-service-ca\") pod \"5221ee38-495a-4653-beb1-df79c0c6d2fd\" (UID: \"5221ee38-495a-4653-beb1-df79c0c6d2fd\") " Sep 06 06:27:47.145254 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:47.145231 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5221ee38-495a-4653-beb1-df79c0c6d2fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5221ee38-495a-4653-beb1-df79c0c6d2fd" (UID: "5221ee38-495a-4653-beb1-df79c0c6d2fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:27:47.245835 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:47.245761 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5221ee38-495a-4653-beb1-df79c0c6d2fd-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:47.932950 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:47.932922 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt" Sep 06 06:27:47.933099 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:47.932897 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt" event={"ID":"5221ee38-495a-4653-beb1-df79c0c6d2fd","Type":"ContainerDied","Data":"da60749d2e4ab9d2dbbd1321a5d1361eceefd3a403faad7482b5f98e5e4d5c57"} Sep 06 06:27:47.933099 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:47.933052 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da60749d2e4ab9d2dbbd1321a5d1361eceefd3a403faad7482b5f98e5e4d5c57" Sep 06 06:27:49.331581 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.331557 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:27:49.468105 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468019 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-workload-certs\") pod \"62e1f506-f005-49aa-8287-d3930a74f0eb\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " Sep 06 06:27:49.468105 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468070 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-data\") pod \"62e1f506-f005-49aa-8287-d3930a74f0eb\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " Sep 06 06:27:49.468336 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468124 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-podinfo\") pod \"62e1f506-f005-49aa-8287-d3930a74f0eb\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " Sep 06 06:27:49.468336 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468143 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-token\") pod \"62e1f506-f005-49aa-8287-d3930a74f0eb\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " Sep 06 06:27:49.468336 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468156 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-envoy\") pod \"62e1f506-f005-49aa-8287-d3930a74f0eb\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " Sep 06 06:27:49.468336 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468210 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-credential-socket\") pod \"62e1f506-f005-49aa-8287-d3930a74f0eb\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " Sep 06 06:27:49.468336 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468236 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62e1f506-f005-49aa-8287-d3930a74f0eb-istiod-ca-cert\") pod \"62e1f506-f005-49aa-8287-d3930a74f0eb\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " Sep 06 06:27:49.468336 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468280 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbm9k\" (UniqueName: \"kubernetes.io/projected/62e1f506-f005-49aa-8287-d3930a74f0eb-kube-api-access-nbm9k\") pod \"62e1f506-f005-49aa-8287-d3930a74f0eb\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " Sep 06 06:27:49.468336 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468307 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-workload-socket\") pod \"62e1f506-f005-49aa-8287-d3930a74f0eb\" (UID: \"62e1f506-f005-49aa-8287-d3930a74f0eb\") " Sep 06 06:27:49.468336 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468326 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "62e1f506-f005-49aa-8287-d3930a74f0eb" (UID: "62e1f506-f005-49aa-8287-d3930a74f0eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:27:49.468730 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468528 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "62e1f506-f005-49aa-8287-d3930a74f0eb" (UID: "62e1f506-f005-49aa-8287-d3930a74f0eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:27:49.468730 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468550 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "62e1f506-f005-49aa-8287-d3930a74f0eb" (UID: "62e1f506-f005-49aa-8287-d3930a74f0eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:27:49.468730 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468692 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:49.468730 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468702 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "62e1f506-f005-49aa-8287-d3930a74f0eb" (UID: "62e1f506-f005-49aa-8287-d3930a74f0eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:27:49.468730 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468709 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:49.469173 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468748 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:49.469173 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.468786 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e1f506-f005-49aa-8287-d3930a74f0eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "62e1f506-f005-49aa-8287-d3930a74f0eb" (UID: "62e1f506-f005-49aa-8287-d3930a74f0eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:27:49.470495 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.470465 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "62e1f506-f005-49aa-8287-d3930a74f0eb" (UID: "62e1f506-f005-49aa-8287-d3930a74f0eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:27:49.470495 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.470475 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e1f506-f005-49aa-8287-d3930a74f0eb-kube-api-access-nbm9k" (OuterVolumeSpecName: "kube-api-access-nbm9k") pod "62e1f506-f005-49aa-8287-d3930a74f0eb" (UID: "62e1f506-f005-49aa-8287-d3930a74f0eb"). InnerVolumeSpecName "kube-api-access-nbm9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:27:49.470679 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.470527 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "62e1f506-f005-49aa-8287-d3930a74f0eb" (UID: "62e1f506-f005-49aa-8287-d3930a74f0eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:27:49.470679 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.470621 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "62e1f506-f005-49aa-8287-d3930a74f0eb" (UID: "62e1f506-f005-49aa-8287-d3930a74f0eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:27:49.570330 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.570299 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:49.570330 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.570326 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:49.570330 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.570334 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:49.570540 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.570341 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62e1f506-f005-49aa-8287-d3930a74f0eb-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:49.570540 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.570349 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbm9k\" (UniqueName: \"kubernetes.io/projected/62e1f506-f005-49aa-8287-d3930a74f0eb-kube-api-access-nbm9k\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:49.570540 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.570359 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62e1f506-f005-49aa-8287-d3930a74f0eb-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:27:49.944999 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.944967 2578 generic.go:356] "Generic (PLEG): container finished" podID="62e1f506-f005-49aa-8287-d3930a74f0eb" containerID="1e2b06e3678853b507acaae15f05e69fbd5ecf11d8f7a290cffab65d041b3e70" exitCode=0 Sep 06 06:27:49.945207 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.945007 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" event={"ID":"62e1f506-f005-49aa-8287-d3930a74f0eb","Type":"ContainerDied","Data":"1e2b06e3678853b507acaae15f05e69fbd5ecf11d8f7a290cffab65d041b3e70"} Sep 06 06:27:49.945207 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.945018 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" Sep 06 06:27:49.945207 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.945057 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k" event={"ID":"62e1f506-f005-49aa-8287-d3930a74f0eb","Type":"ContainerDied","Data":"346df172ea6c920dfa47e436f0f624de56ba22f3bb49db1871e98c3e96633e11"} Sep 06 06:27:49.945207 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.945088 2578 scope.go:117] "RemoveContainer" containerID="1e2b06e3678853b507acaae15f05e69fbd5ecf11d8f7a290cffab65d041b3e70" Sep 06 06:27:49.954506 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.954488 2578 scope.go:117] "RemoveContainer" containerID="40dd2af1eefcbdb8782ad30dc5f19403cff8db4303f1b8fb3d74da0f8d807968" Sep 06 06:27:49.963296 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.963229 2578 scope.go:117] "RemoveContainer" containerID="1e2b06e3678853b507acaae15f05e69fbd5ecf11d8f7a290cffab65d041b3e70" Sep 06 06:27:49.963720 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:27:49.963693 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2b06e3678853b507acaae15f05e69fbd5ecf11d8f7a290cffab65d041b3e70\": container with ID starting with 1e2b06e3678853b507acaae15f05e69fbd5ecf11d8f7a290cffab65d041b3e70 not found: ID does not exist" containerID="1e2b06e3678853b507acaae15f05e69fbd5ecf11d8f7a290cffab65d041b3e70" Sep 06 06:27:49.963828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.963732 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2b06e3678853b507acaae15f05e69fbd5ecf11d8f7a290cffab65d041b3e70"} err="failed to get container status \"1e2b06e3678853b507acaae15f05e69fbd5ecf11d8f7a290cffab65d041b3e70\": rpc error: code = NotFound desc = could not find container \"1e2b06e3678853b507acaae15f05e69fbd5ecf11d8f7a290cffab65d041b3e70\": container with ID starting with 1e2b06e3678853b507acaae15f05e69fbd5ecf11d8f7a290cffab65d041b3e70 not found: ID does not exist" Sep 06 06:27:49.963828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.963757 2578 scope.go:117] "RemoveContainer" containerID="40dd2af1eefcbdb8782ad30dc5f19403cff8db4303f1b8fb3d74da0f8d807968" Sep 06 06:27:49.964095 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:27:49.964077 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40dd2af1eefcbdb8782ad30dc5f19403cff8db4303f1b8fb3d74da0f8d807968\": container with ID starting with 40dd2af1eefcbdb8782ad30dc5f19403cff8db4303f1b8fb3d74da0f8d807968 not found: ID does not exist" containerID="40dd2af1eefcbdb8782ad30dc5f19403cff8db4303f1b8fb3d74da0f8d807968" Sep 06 06:27:49.964151 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.964101 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40dd2af1eefcbdb8782ad30dc5f19403cff8db4303f1b8fb3d74da0f8d807968"} err="failed to get container status \"40dd2af1eefcbdb8782ad30dc5f19403cff8db4303f1b8fb3d74da0f8d807968\": rpc error: code = NotFound desc = could not find container \"40dd2af1eefcbdb8782ad30dc5f19403cff8db4303f1b8fb3d74da0f8d807968\": container with ID starting with 40dd2af1eefcbdb8782ad30dc5f19403cff8db4303f1b8fb3d74da0f8d807968 not found: ID does not exist" Sep 06 06:27:49.965766 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.965748 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k"] Sep 06 06:27:49.968729 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:49.968708 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a52b50-openshift-default-d9577795d-r4g7k"] Sep 06 06:27:51.145957 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:51.145920 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62e1f506-f005-49aa-8287-d3930a74f0eb" path="/var/lib/kubelet/pods/62e1f506-f005-49aa-8287-d3930a74f0eb/volumes" Sep 06 06:27:55.107104 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:55.107067 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt"] Sep 06 06:27:55.110026 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:55.109996 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-429831-c7wlt"] Sep 06 06:27:55.144712 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:55.144684 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5221ee38-495a-4653-beb1-df79c0c6d2fd" path="/var/lib/kubelet/pods/5221ee38-495a-4653-beb1-df79c0c6d2fd/volumes" Sep 06 06:27:57.982302 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:27:57.982266 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:28:01.995259 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:01.995226 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq"] Sep 06 06:28:01.995681 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:01.995581 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e1f506-f005-49aa-8287-d3930a74f0eb" containerName="istio-proxy" Sep 06 06:28:01.995681 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:01.995592 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e1f506-f005-49aa-8287-d3930a74f0eb" containerName="istio-proxy" Sep 06 06:28:01.995681 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:01.995613 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5221ee38-495a-4653-beb1-df79c0c6d2fd" containerName="revoke-keys" Sep 06 06:28:01.995681 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:01.995619 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5221ee38-495a-4653-beb1-df79c0c6d2fd" containerName="revoke-keys" Sep 06 06:28:01.995681 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:01.995625 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e1f506-f005-49aa-8287-d3930a74f0eb" containerName="istio-proxy" Sep 06 06:28:01.995681 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:01.995631 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e1f506-f005-49aa-8287-d3930a74f0eb" containerName="istio-proxy" Sep 06 06:28:01.995929 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:01.995687 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="62e1f506-f005-49aa-8287-d3930a74f0eb" containerName="istio-proxy" Sep 06 06:28:01.995929 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:01.995696 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5221ee38-495a-4653-beb1-df79c0c6d2fd" containerName="revoke-keys" Sep 06 06:28:01.995929 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:01.995703 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="62e1f506-f005-49aa-8287-d3930a74f0eb" containerName="istio-proxy" Sep 06 06:28:01.998896 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:01.998878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq" Sep 06 06:28:02.000939 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:02.000886 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:28:02.001045 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:02.000916 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mrdjq\"" Sep 06 06:28:02.006918 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:02.006278 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq"] Sep 06 06:28:02.095240 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:02.095206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0098b683-01eb-4d71-bbad-e1672dbb7910-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq\" (UID: \"0098b683-01eb-4d71-bbad-e1672dbb7910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq" Sep 06 06:28:02.196735 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:02.196698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0098b683-01eb-4d71-bbad-e1672dbb7910-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq\" (UID: \"0098b683-01eb-4d71-bbad-e1672dbb7910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq" Sep 06 06:28:02.197321 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:02.197301 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0098b683-01eb-4d71-bbad-e1672dbb7910-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq\" (UID: \"0098b683-01eb-4d71-bbad-e1672dbb7910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq" Sep 06 06:28:02.310862 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:02.310829 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq" Sep 06 06:28:02.578681 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:02.578655 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq"] Sep 06 06:28:02.580847 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:28:02.580818 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0098b683_01eb_4d71_bbad_e1672dbb7910.slice/crio-5d81cf03265865a5b3db02280cf69e320765a86a8954f2504fa9e4b3f86c5c8a WatchSource:0}: Error finding container 5d81cf03265865a5b3db02280cf69e320765a86a8954f2504fa9e4b3f86c5c8a: Status 404 returned error can't find the container with id 5d81cf03265865a5b3db02280cf69e320765a86a8954f2504fa9e4b3f86c5c8a Sep 06 06:28:03.018830 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:03.018791 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq" event={"ID":"0098b683-01eb-4d71-bbad-e1672dbb7910","Type":"ContainerStarted","Data":"0eaf3d97bbb66a2160420f8d2fe01701109564ffd573d106f70374a9d4accffb"} Sep 06 06:28:03.019226 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:03.018846 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq" event={"ID":"0098b683-01eb-4d71-bbad-e1672dbb7910","Type":"ContainerStarted","Data":"5d81cf03265865a5b3db02280cf69e320765a86a8954f2504fa9e4b3f86c5c8a"} Sep 06 06:28:03.033375 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:03.033327 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq" podStartSLOduration=2.033311737 podStartE2EDuration="2.033311737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:28:03.031884868 +0000 UTC m=+2048.430215337" watchObservedRunningTime="2026-09-06 06:28:03.033311737 +0000 UTC m=+2048.431642203" Sep 06 06:28:04.025148 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:04.025113 2578 generic.go:356] "Generic (PLEG): container finished" podID="0098b683-01eb-4d71-bbad-e1672dbb7910" containerID="0eaf3d97bbb66a2160420f8d2fe01701109564ffd573d106f70374a9d4accffb" exitCode=0 Sep 06 06:28:04.025541 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:04.025210 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq" event={"ID":"0098b683-01eb-4d71-bbad-e1672dbb7910","Type":"ContainerDied","Data":"0eaf3d97bbb66a2160420f8d2fe01701109564ffd573d106f70374a9d4accffb"} Sep 06 06:28:05.174753 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:05.174727 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq" Sep 06 06:28:05.227172 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:05.227147 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0098b683-01eb-4d71-bbad-e1672dbb7910-maas-api-service-ca\") pod \"0098b683-01eb-4d71-bbad-e1672dbb7910\" (UID: \"0098b683-01eb-4d71-bbad-e1672dbb7910\") " Sep 06 06:28:05.227499 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:05.227479 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0098b683-01eb-4d71-bbad-e1672dbb7910-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0098b683-01eb-4d71-bbad-e1672dbb7910" (UID: "0098b683-01eb-4d71-bbad-e1672dbb7910"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:28:05.328526 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:05.328440 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0098b683-01eb-4d71-bbad-e1672dbb7910-maas-api-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:28:06.036147 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:06.036116 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq" Sep 06 06:28:06.036147 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:06.036140 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq" event={"ID":"0098b683-01eb-4d71-bbad-e1672dbb7910","Type":"ContainerDied","Data":"5d81cf03265865a5b3db02280cf69e320765a86a8954f2504fa9e4b3f86c5c8a"} Sep 06 06:28:06.036359 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:06.036186 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d81cf03265865a5b3db02280cf69e320765a86a8954f2504fa9e4b3f86c5c8a" Sep 06 06:28:12.597121 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:12.597081 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq"] Sep 06 06:28:12.599640 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:12.599615 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dc97f9-wbphq"] Sep 06 06:28:13.145288 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:13.145256 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0098b683-01eb-4d71-bbad-e1672dbb7910" path="/var/lib/kubelet/pods/0098b683-01eb-4d71-bbad-e1672dbb7910/volumes" Sep 06 06:28:14.973460 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:14.973427 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:28:18.957745 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:18.957663 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7"] Sep 06 06:28:18.958189 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:18.957872 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" podUID="6f296ad6-71a1-4e88-99bb-62b836296b7d" containerName="istio-proxy" containerID="cri-o://c60dab4dea7b275d93c937406fdcfe7b0a0a0b2b3516b25b8b14dc723a8274ef" gracePeriod=30 Sep 06 06:28:20.558577 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:20.558535 2578 patch_prober.go:28] interesting pod/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 06 06:28:20.558984 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:20.558593 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" podUID="6f296ad6-71a1-4e88-99bb-62b836296b7d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 06 06:28:24.073457 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.073430 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:28:24.131726 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.131698 2578 generic.go:356] "Generic (PLEG): container finished" podID="6f296ad6-71a1-4e88-99bb-62b836296b7d" containerID="c60dab4dea7b275d93c937406fdcfe7b0a0a0b2b3516b25b8b14dc723a8274ef" exitCode=0 Sep 06 06:28:24.131891 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.131772 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" event={"ID":"6f296ad6-71a1-4e88-99bb-62b836296b7d","Type":"ContainerDied","Data":"c60dab4dea7b275d93c937406fdcfe7b0a0a0b2b3516b25b8b14dc723a8274ef"} Sep 06 06:28:24.131891 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.131828 2578 scope.go:117] "RemoveContainer" containerID="0612a1bf22de3c96f48fc79f347062a4d764dd253e19bbc08ab942d433ca1941" Sep 06 06:28:24.210024 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.210000 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:28:24.302384 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.302358 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-data\") pod \"6f296ad6-71a1-4e88-99bb-62b836296b7d\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " Sep 06 06:28:24.302544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.302392 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f296ad6-71a1-4e88-99bb-62b836296b7d-istiod-ca-cert\") pod \"6f296ad6-71a1-4e88-99bb-62b836296b7d\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " Sep 06 06:28:24.302544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.302446 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-workload-socket\") pod \"6f296ad6-71a1-4e88-99bb-62b836296b7d\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " Sep 06 06:28:24.302544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.302475 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-token\") pod \"6f296ad6-71a1-4e88-99bb-62b836296b7d\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " Sep 06 06:28:24.302544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.302495 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsh29\" (UniqueName: \"kubernetes.io/projected/6f296ad6-71a1-4e88-99bb-62b836296b7d-kube-api-access-fsh29\") pod \"6f296ad6-71a1-4e88-99bb-62b836296b7d\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " Sep 06 06:28:24.302544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.302511 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-podinfo\") pod \"6f296ad6-71a1-4e88-99bb-62b836296b7d\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " Sep 06 06:28:24.302829 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.302605 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-credential-socket\") pod \"6f296ad6-71a1-4e88-99bb-62b836296b7d\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " Sep 06 06:28:24.302829 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.302652 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-workload-certs\") pod \"6f296ad6-71a1-4e88-99bb-62b836296b7d\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " Sep 06 06:28:24.302829 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.302674 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f296ad6-71a1-4e88-99bb-62b836296b7d" (UID: "6f296ad6-71a1-4e88-99bb-62b836296b7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:28:24.302829 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.302698 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-envoy\") pod \"6f296ad6-71a1-4e88-99bb-62b836296b7d\" (UID: \"6f296ad6-71a1-4e88-99bb-62b836296b7d\") " Sep 06 06:28:24.302829 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.302757 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f296ad6-71a1-4e88-99bb-62b836296b7d" (UID: "6f296ad6-71a1-4e88-99bb-62b836296b7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:28:24.303086 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.302880 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f296ad6-71a1-4e88-99bb-62b836296b7d" (UID: "6f296ad6-71a1-4e88-99bb-62b836296b7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:28:24.303086 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.302945 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f296ad6-71a1-4e88-99bb-62b836296b7d" (UID: "6f296ad6-71a1-4e88-99bb-62b836296b7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:28:24.303177 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.303100 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-credential-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:28:24.303177 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.303120 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-workload-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:28:24.303177 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.303134 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-data\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:28:24.303177 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.303146 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-workload-socket\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:28:24.303599 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.303573 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f296ad6-71a1-4e88-99bb-62b836296b7d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f296ad6-71a1-4e88-99bb-62b836296b7d" (UID: "6f296ad6-71a1-4e88-99bb-62b836296b7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:28:24.304781 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.304755 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f296ad6-71a1-4e88-99bb-62b836296b7d" (UID: "6f296ad6-71a1-4e88-99bb-62b836296b7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:28:24.304890 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.304848 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f296ad6-71a1-4e88-99bb-62b836296b7d-kube-api-access-fsh29" (OuterVolumeSpecName: "kube-api-access-fsh29") pod "6f296ad6-71a1-4e88-99bb-62b836296b7d" (UID: "6f296ad6-71a1-4e88-99bb-62b836296b7d"). InnerVolumeSpecName "kube-api-access-fsh29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:28:24.304974 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.304941 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f296ad6-71a1-4e88-99bb-62b836296b7d" (UID: "6f296ad6-71a1-4e88-99bb-62b836296b7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 06 06:28:24.305472 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.305446 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f296ad6-71a1-4e88-99bb-62b836296b7d" (UID: "6f296ad6-71a1-4e88-99bb-62b836296b7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 06 06:28:24.404321 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.404287 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsh29\" (UniqueName: \"kubernetes.io/projected/6f296ad6-71a1-4e88-99bb-62b836296b7d-kube-api-access-fsh29\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:28:24.404321 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.404318 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-podinfo\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:28:24.404507 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.404334 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-envoy\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:28:24.404507 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.404345 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f296ad6-71a1-4e88-99bb-62b836296b7d-istiod-ca-cert\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:28:24.404507 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:24.404356 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f296ad6-71a1-4e88-99bb-62b836296b7d-istio-token\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:28:25.138107 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:25.138065 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" event={"ID":"6f296ad6-71a1-4e88-99bb-62b836296b7d","Type":"ContainerDied","Data":"758913b6964646d830763519dbf6708cc6b597543b13985e723cad6c95d88d17"} Sep 06 06:28:25.138544 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:25.138207 2578 scope.go:117] "RemoveContainer" containerID="c60dab4dea7b275d93c937406fdcfe7b0a0a0b2b3516b25b8b14dc723a8274ef" Sep 06 06:28:25.140631 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:25.140609 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7" Sep 06 06:28:25.164866 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:25.164824 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7"] Sep 06 06:28:25.169423 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:25.169392 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-dc97f9-openshift-default-6bd9d4bcc4-8wcd7"] Sep 06 06:28:27.149711 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:27.146566 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f296ad6-71a1-4e88-99bb-62b836296b7d" path="/var/lib/kubelet/pods/6f296ad6-71a1-4e88-99bb-62b836296b7d/volumes" Sep 06 06:28:55.232651 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:55.232620 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 06:28:55.238624 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:55.238600 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 06:28:55.687288 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:55.687252 2578 scope.go:117] "RemoveContainer" containerID="61babb0a2bd1e9c4fccd0142ab7c5710260c75ba418c01a80d5f032111998900" Sep 06 06:28:55.696843 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:28:55.696817 2578 scope.go:117] "RemoveContainer" containerID="cfa26524bf4aec1860efaa83f58897ae2ba8c93f450d45bc3f0cbc96825b5c40" Sep 06 06:29:44.380218 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:29:44.380186 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:29:55.723313 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:29:55.723240 2578 scope.go:117] "RemoveContainer" containerID="ccc5a6e9c8e707ebe6ff516a1af1362e1dc8052dce12d2509910ed328d65f7f7" Sep 06 06:29:55.731326 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:29:55.731306 2578 scope.go:117] "RemoveContainer" containerID="7467c641ad4ea318b4dfb9f2ad33ea3b80717569d6108cd4951ed79eb0832504" Sep 06 06:30:08.874531 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:30:08.874487 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:30:16.781283 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:30:16.781247 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:30:24.874543 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:30:24.874507 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:30:44.877431 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:30:44.877394 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:30:52.603780 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:30:52.603750 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:32:24.193371 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:32:24.193338 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:32:31.087354 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:32:31.087316 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:32:39.574455 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:32:39.574409 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:33:02.079410 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:02.079323 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:33:30.684841 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:30.684804 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:33:37.975251 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:37.975208 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:33:45.303426 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.303389 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-76f875657b-84b84"] Sep 06 06:33:45.304003 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.303914 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f296ad6-71a1-4e88-99bb-62b836296b7d" containerName="istio-proxy" Sep 06 06:33:45.304003 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.303935 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f296ad6-71a1-4e88-99bb-62b836296b7d" containerName="istio-proxy" Sep 06 06:33:45.304003 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.303949 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f296ad6-71a1-4e88-99bb-62b836296b7d" containerName="istio-proxy" Sep 06 06:33:45.304003 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.303955 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f296ad6-71a1-4e88-99bb-62b836296b7d" containerName="istio-proxy" Sep 06 06:33:45.304003 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.303976 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0098b683-01eb-4d71-bbad-e1672dbb7910" containerName="revoke-keys" Sep 06 06:33:45.304003 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.303981 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0098b683-01eb-4d71-bbad-e1672dbb7910" containerName="revoke-keys" Sep 06 06:33:45.304224 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.304055 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f296ad6-71a1-4e88-99bb-62b836296b7d" containerName="istio-proxy" Sep 06 06:33:45.304224 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.304066 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0098b683-01eb-4d71-bbad-e1672dbb7910" containerName="revoke-keys" Sep 06 06:33:45.307162 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.307143 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:45.310408 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.310377 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 06 06:33:45.310520 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.310423 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5vd8t\"" Sep 06 06:33:45.310520 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.310438 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 06 06:33:45.310520 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.310440 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 06 06:33:45.310520 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.310467 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 06 06:33:45.313245 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.313223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/054135de-53da-4531-913b-476262a021bf-metrics-certs\") pod \"maas-api-76f875657b-84b84\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:45.313349 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.313269 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054135de-53da-4531-913b-476262a021bf-maas-api-tls\") pod \"maas-api-76f875657b-84b84\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:45.313412 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.313380 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tfjz\" (UniqueName: \"kubernetes.io/projected/054135de-53da-4531-913b-476262a021bf-kube-api-access-8tfjz\") pod \"maas-api-76f875657b-84b84\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:45.313473 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.313433 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/054135de-53da-4531-913b-476262a021bf-openshift-service-ca\") pod \"maas-api-76f875657b-84b84\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:45.316477 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.316458 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76f875657b-84b84"] Sep 06 06:33:45.413973 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.413942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/054135de-53da-4531-913b-476262a021bf-metrics-certs\") pod \"maas-api-76f875657b-84b84\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:45.414149 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.413983 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054135de-53da-4531-913b-476262a021bf-maas-api-tls\") pod \"maas-api-76f875657b-84b84\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:45.414149 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.414029 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tfjz\" (UniqueName: \"kubernetes.io/projected/054135de-53da-4531-913b-476262a021bf-kube-api-access-8tfjz\") pod \"maas-api-76f875657b-84b84\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:45.414149 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.414060 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/054135de-53da-4531-913b-476262a021bf-openshift-service-ca\") pod \"maas-api-76f875657b-84b84\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:45.414635 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.414610 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/054135de-53da-4531-913b-476262a021bf-openshift-service-ca\") pod \"maas-api-76f875657b-84b84\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:45.416499 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.416476 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054135de-53da-4531-913b-476262a021bf-maas-api-tls\") pod \"maas-api-76f875657b-84b84\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:45.416639 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.416619 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/054135de-53da-4531-913b-476262a021bf-metrics-certs\") pod \"maas-api-76f875657b-84b84\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:45.421088 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.421068 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tfjz\" (UniqueName: \"kubernetes.io/projected/054135de-53da-4531-913b-476262a021bf-kube-api-access-8tfjz\") pod \"maas-api-76f875657b-84b84\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:45.619444 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.619345 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:45.889103 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.889079 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76f875657b-84b84"] Sep 06 06:33:45.891225 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:33:45.891201 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054135de_53da_4531_913b_476262a021bf.slice/crio-cde6fed43869294d641e0fb199464c5236b4ee8eacf5c7fa073b8643da2a9abc WatchSource:0}: Error finding container cde6fed43869294d641e0fb199464c5236b4ee8eacf5c7fa073b8643da2a9abc: Status 404 returned error can't find the container with id cde6fed43869294d641e0fb199464c5236b4ee8eacf5c7fa073b8643da2a9abc Sep 06 06:33:45.892606 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:45.892585 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 06 06:33:46.756842 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:46.756746 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" event={"ID":"054135de-53da-4531-913b-476262a021bf","Type":"ContainerStarted","Data":"6786e89c10b5fc8d15e71633699500c75a6c887670e1afd70cdd16c5271b5820"} Sep 06 06:33:46.757306 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:46.756868 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:46.757306 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:46.756891 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" event={"ID":"054135de-53da-4531-913b-476262a021bf","Type":"ContainerStarted","Data":"cde6fed43869294d641e0fb199464c5236b4ee8eacf5c7fa073b8643da2a9abc"} Sep 06 06:33:46.772515 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:46.772463 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" podStartSLOduration=1.286138343 podStartE2EDuration="1.772446276s" totalImagesPullingTime="486.307933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:33:46.771881437 +0000 UTC m=+2392.170211954" watchObservedRunningTime="2026-09-06 06:33:46.772446276 +0000 UTC m=+2392.170776746" Sep 06 06:33:46.788823 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:46.788788 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:33:52.769086 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:52.769056 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:33:52.885039 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:52.884999 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:33:53.474592 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.474559 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6"] Sep 06 06:33:53.475033 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.475020 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f296ad6-71a1-4e88-99bb-62b836296b7d" containerName="istio-proxy" Sep 06 06:33:53.478166 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.478151 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:33:53.484363 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.484331 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6"] Sep 06 06:33:53.587585 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.587538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwx7f\" (UniqueName: \"kubernetes.io/projected/d17ebd1b-be71-4995-bc5b-e016d4f7a198-kube-api-access-dwx7f\") pod \"maas-api-7c946f6b4d-2qjp6\" (UID: \"d17ebd1b-be71-4995-bc5b-e016d4f7a198\") " pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:33:53.587792 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.587652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d17ebd1b-be71-4995-bc5b-e016d4f7a198-maas-api-tls\") pod \"maas-api-7c946f6b4d-2qjp6\" (UID: \"d17ebd1b-be71-4995-bc5b-e016d4f7a198\") " pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:33:53.587792 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.587682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d17ebd1b-be71-4995-bc5b-e016d4f7a198-metrics-certs\") pod \"maas-api-7c946f6b4d-2qjp6\" (UID: \"d17ebd1b-be71-4995-bc5b-e016d4f7a198\") " pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:33:53.587792 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.587720 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17ebd1b-be71-4995-bc5b-e016d4f7a198-openshift-service-ca\") pod \"maas-api-7c946f6b4d-2qjp6\" (UID: \"d17ebd1b-be71-4995-bc5b-e016d4f7a198\") " pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:33:53.688525 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.688475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17ebd1b-be71-4995-bc5b-e016d4f7a198-openshift-service-ca\") pod \"maas-api-7c946f6b4d-2qjp6\" (UID: \"d17ebd1b-be71-4995-bc5b-e016d4f7a198\") " pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:33:53.688743 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.688564 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwx7f\" (UniqueName: \"kubernetes.io/projected/d17ebd1b-be71-4995-bc5b-e016d4f7a198-kube-api-access-dwx7f\") pod \"maas-api-7c946f6b4d-2qjp6\" (UID: \"d17ebd1b-be71-4995-bc5b-e016d4f7a198\") " pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:33:53.688743 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.688670 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d17ebd1b-be71-4995-bc5b-e016d4f7a198-maas-api-tls\") pod \"maas-api-7c946f6b4d-2qjp6\" (UID: \"d17ebd1b-be71-4995-bc5b-e016d4f7a198\") " pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:33:53.688743 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.688693 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d17ebd1b-be71-4995-bc5b-e016d4f7a198-metrics-certs\") pod \"maas-api-7c946f6b4d-2qjp6\" (UID: \"d17ebd1b-be71-4995-bc5b-e016d4f7a198\") " pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:33:53.689290 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.689267 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17ebd1b-be71-4995-bc5b-e016d4f7a198-openshift-service-ca\") pod \"maas-api-7c946f6b4d-2qjp6\" (UID: \"d17ebd1b-be71-4995-bc5b-e016d4f7a198\") " pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:33:53.691465 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.691440 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d17ebd1b-be71-4995-bc5b-e016d4f7a198-metrics-certs\") pod \"maas-api-7c946f6b4d-2qjp6\" (UID: \"d17ebd1b-be71-4995-bc5b-e016d4f7a198\") " pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:33:53.691465 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.691458 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d17ebd1b-be71-4995-bc5b-e016d4f7a198-maas-api-tls\") pod \"maas-api-7c946f6b4d-2qjp6\" (UID: \"d17ebd1b-be71-4995-bc5b-e016d4f7a198\") " pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:33:53.696432 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.696401 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwx7f\" (UniqueName: \"kubernetes.io/projected/d17ebd1b-be71-4995-bc5b-e016d4f7a198-kube-api-access-dwx7f\") pod \"maas-api-7c946f6b4d-2qjp6\" (UID: \"d17ebd1b-be71-4995-bc5b-e016d4f7a198\") " pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:33:53.790842 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:53.790806 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:33:54.068765 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:54.065058 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6"] Sep 06 06:33:54.071396 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:33:54.071369 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17ebd1b_be71_4995_bc5b_e016d4f7a198.slice/crio-f4464b2d7e5c6b8d8af175aa88692fa470014e38b07359680b84023b9227c954 WatchSource:0}: Error finding container f4464b2d7e5c6b8d8af175aa88692fa470014e38b07359680b84023b9227c954: Status 404 returned error can't find the container with id f4464b2d7e5c6b8d8af175aa88692fa470014e38b07359680b84023b9227c954 Sep 06 06:33:54.799891 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:54.799854 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" event={"ID":"d17ebd1b-be71-4995-bc5b-e016d4f7a198","Type":"ContainerStarted","Data":"b45fd9f0e849c2696057828738ee0d19a1df0ee4c5dd745077be12ec355a7ce9"} Sep 06 06:33:54.800331 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:54.799946 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:33:54.800331 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:54.799972 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" event={"ID":"d17ebd1b-be71-4995-bc5b-e016d4f7a198","Type":"ContainerStarted","Data":"f4464b2d7e5c6b8d8af175aa88692fa470014e38b07359680b84023b9227c954"} Sep 06 06:33:54.818302 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:54.818256 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" podStartSLOduration=1.469133727 podStartE2EDuration="1.818240567s" totalImagesPullingTime="349.10684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:33:54.817181808 +0000 UTC m=+2400.215512305" watchObservedRunningTime="2026-09-06 06:33:54.818240567 +0000 UTC m=+2400.216571035" Sep 06 06:33:55.264387 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:55.264298 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 06:33:55.271828 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:55.271800 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 06:33:55.774541 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:55.774508 2578 scope.go:117] "RemoveContainer" containerID="d0ff7108ce3727d3717eb58f0c0f4f7b88990de22016f810adb3ff27f8dbf28a" Sep 06 06:33:59.883084 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:33:59.883050 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:34:00.811589 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:00.811556 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c946f6b4d-2qjp6" Sep 06 06:34:00.854803 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:00.854767 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-76f875657b-84b84"] Sep 06 06:34:00.855041 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:00.855014 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" podUID="054135de-53da-4531-913b-476262a021bf" containerName="maas-api" containerID="cri-o://6786e89c10b5fc8d15e71633699500c75a6c887670e1afd70cdd16c5271b5820" gracePeriod=30 Sep 06 06:34:02.099265 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.099240 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:34:02.168314 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.168216 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/054135de-53da-4531-913b-476262a021bf-metrics-certs\") pod \"054135de-53da-4531-913b-476262a021bf\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " Sep 06 06:34:02.168314 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.168261 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054135de-53da-4531-913b-476262a021bf-maas-api-tls\") pod \"054135de-53da-4531-913b-476262a021bf\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " Sep 06 06:34:02.168551 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.168325 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/054135de-53da-4531-913b-476262a021bf-openshift-service-ca\") pod \"054135de-53da-4531-913b-476262a021bf\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " Sep 06 06:34:02.168551 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.168355 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tfjz\" (UniqueName: \"kubernetes.io/projected/054135de-53da-4531-913b-476262a021bf-kube-api-access-8tfjz\") pod \"054135de-53da-4531-913b-476262a021bf\" (UID: \"054135de-53da-4531-913b-476262a021bf\") " Sep 06 06:34:02.168718 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.168688 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054135de-53da-4531-913b-476262a021bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "054135de-53da-4531-913b-476262a021bf" (UID: "054135de-53da-4531-913b-476262a021bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 06 06:34:02.170547 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.170514 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054135de-53da-4531-913b-476262a021bf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "054135de-53da-4531-913b-476262a021bf" (UID: "054135de-53da-4531-913b-476262a021bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:34:02.170638 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.170517 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054135de-53da-4531-913b-476262a021bf-kube-api-access-8tfjz" (OuterVolumeSpecName: "kube-api-access-8tfjz") pod "054135de-53da-4531-913b-476262a021bf" (UID: "054135de-53da-4531-913b-476262a021bf"). InnerVolumeSpecName "kube-api-access-8tfjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 06 06:34:02.170638 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.170569 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054135de-53da-4531-913b-476262a021bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "054135de-53da-4531-913b-476262a021bf" (UID: "054135de-53da-4531-913b-476262a021bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 06 06:34:02.269548 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.269506 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/054135de-53da-4531-913b-476262a021bf-openshift-service-ca\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:34:02.269548 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.269541 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tfjz\" (UniqueName: \"kubernetes.io/projected/054135de-53da-4531-913b-476262a021bf-kube-api-access-8tfjz\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:34:02.269548 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.269553 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/054135de-53da-4531-913b-476262a021bf-metrics-certs\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:34:02.269801 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.269561 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/054135de-53da-4531-913b-476262a021bf-maas-api-tls\") on node \"ip-10-0-135-32.ec2.internal\" DevicePath \"\"" Sep 06 06:34:02.841706 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.841666 2578 generic.go:356] "Generic (PLEG): container finished" podID="054135de-53da-4531-913b-476262a021bf" containerID="6786e89c10b5fc8d15e71633699500c75a6c887670e1afd70cdd16c5271b5820" exitCode=0 Sep 06 06:34:02.841927 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.841723 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" Sep 06 06:34:02.841927 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.841756 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" event={"ID":"054135de-53da-4531-913b-476262a021bf","Type":"ContainerDied","Data":"6786e89c10b5fc8d15e71633699500c75a6c887670e1afd70cdd16c5271b5820"} Sep 06 06:34:02.841927 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.841812 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76f875657b-84b84" event={"ID":"054135de-53da-4531-913b-476262a021bf","Type":"ContainerDied","Data":"cde6fed43869294d641e0fb199464c5236b4ee8eacf5c7fa073b8643da2a9abc"} Sep 06 06:34:02.841927 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.841834 2578 scope.go:117] "RemoveContainer" containerID="6786e89c10b5fc8d15e71633699500c75a6c887670e1afd70cdd16c5271b5820" Sep 06 06:34:02.851371 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.851355 2578 scope.go:117] "RemoveContainer" containerID="6786e89c10b5fc8d15e71633699500c75a6c887670e1afd70cdd16c5271b5820" Sep 06 06:34:02.851664 ip-10-0-135-32 kubenswrapper[2578]: E0906 06:34:02.851643 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6786e89c10b5fc8d15e71633699500c75a6c887670e1afd70cdd16c5271b5820\": container with ID starting with 6786e89c10b5fc8d15e71633699500c75a6c887670e1afd70cdd16c5271b5820 not found: ID does not exist" containerID="6786e89c10b5fc8d15e71633699500c75a6c887670e1afd70cdd16c5271b5820" Sep 06 06:34:02.851750 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.851675 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6786e89c10b5fc8d15e71633699500c75a6c887670e1afd70cdd16c5271b5820"} err="failed to get container status \"6786e89c10b5fc8d15e71633699500c75a6c887670e1afd70cdd16c5271b5820\": rpc error: code = NotFound desc = could not find container \"6786e89c10b5fc8d15e71633699500c75a6c887670e1afd70cdd16c5271b5820\": container with ID starting with 6786e89c10b5fc8d15e71633699500c75a6c887670e1afd70cdd16c5271b5820 not found: ID does not exist" Sep 06 06:34:02.861224 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.861200 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-76f875657b-84b84"] Sep 06 06:34:02.864658 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:02.864636 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-76f875657b-84b84"] Sep 06 06:34:03.144871 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:03.144790 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="054135de-53da-4531-913b-476262a021bf" path="/var/lib/kubelet/pods/054135de-53da-4531-913b-476262a021bf/volumes" Sep 06 06:34:05.774802 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:05.774766 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:34:13.780730 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:13.780684 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-hzz62"] Sep 06 06:34:20.555720 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:20.555639 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-ssg8b_63866f27-a454-4498-b1ab-14973b58a81e/manager/0.log" Sep 06 06:34:20.910290 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:20.910206 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-xslf6_c525d127-0f22-4fc8-84e8-58a618a0dbaa/manager/0.log" Sep 06 06:34:21.262270 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:21.262187 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b897cd959-rsx7c_3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266/copy-manifests/0.log" Sep 06 06:34:21.268520 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:21.268496 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b897cd959-rsx7c_3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266/manager/0.log" Sep 06 06:34:21.589898 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:21.589853 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c946f6b4d-2qjp6_d17ebd1b-be71-4995-bc5b-e016d4f7a198/maas-api/0.log" Sep 06 06:34:21.805231 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:21.805202 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29811255-5ml4h_b61512e6-1f66-4c8b-8b17-d751660cda38/cleanup/0.log" Sep 06 06:34:23.103463 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:23.103437 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-zsfs2_76d99e41-d9a6-4a88-8960-00637b5b6efa/kuadrant-console-plugin/0.log" Sep 06 06:34:23.331624 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:23.331593 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-hzz62_89f58454-2203-4916-9a9f-76c089e3dcba/limitador/0.log" Sep 06 06:34:23.444018 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:23.443946 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-9kxg6_3c20a402-67b1-447a-91cd-2ef5c2d39255/manager/0.log" Sep 06 06:34:23.783152 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:23.783118 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv_9881c8ee-3e68-4e99-a17c-198451cbf660/istio-proxy/0.log" Sep 06 06:34:24.226043 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:24.225948 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rkjhl_b1e56087-4754-48ce-9c1c-8f16b57c74b1/istio-proxy/1.log" Sep 06 06:34:24.565370 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:24.565340 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-99f956f9b-9d6ht_c9644473-eedd-4380-a23d-ecf5b0c85f10/router/0.log" Sep 06 06:34:25.346450 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:25.346414 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkwms6_ba09e890-672a-49c5-a57a-c44d16e07c1d/main/0.log" Sep 06 06:34:31.067718 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.067681 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zzhm6/must-gather-vwt4f"] Sep 06 06:34:31.068123 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.068090 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054135de-53da-4531-913b-476262a021bf" containerName="maas-api" Sep 06 06:34:31.068123 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.068102 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="054135de-53da-4531-913b-476262a021bf" containerName="maas-api" Sep 06 06:34:31.068197 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.068162 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="054135de-53da-4531-913b-476262a021bf" containerName="maas-api" Sep 06 06:34:31.071192 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.071172 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzhm6/must-gather-vwt4f" Sep 06 06:34:31.073614 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.073593 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zzhm6\"/\"kube-root-ca.crt\"" Sep 06 06:34:31.074094 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.074075 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zzhm6\"/\"openshift-service-ca.crt\"" Sep 06 06:34:31.074174 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.074076 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zzhm6\"/\"default-dockercfg-gv2n8\"" Sep 06 06:34:31.078640 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.078621 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzhm6/must-gather-vwt4f"] Sep 06 06:34:31.152946 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.152892 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/187d9d65-8478-4d4a-b79c-70f1555af1ee-must-gather-output\") pod \"must-gather-vwt4f\" (UID: \"187d9d65-8478-4d4a-b79c-70f1555af1ee\") " pod="openshift-must-gather-zzhm6/must-gather-vwt4f" Sep 06 06:34:31.153147 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.152977 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkkpt\" (UniqueName: \"kubernetes.io/projected/187d9d65-8478-4d4a-b79c-70f1555af1ee-kube-api-access-xkkpt\") pod \"must-gather-vwt4f\" (UID: \"187d9d65-8478-4d4a-b79c-70f1555af1ee\") " pod="openshift-must-gather-zzhm6/must-gather-vwt4f" Sep 06 06:34:31.254106 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.254066 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/187d9d65-8478-4d4a-b79c-70f1555af1ee-must-gather-output\") pod \"must-gather-vwt4f\" (UID: \"187d9d65-8478-4d4a-b79c-70f1555af1ee\") " pod="openshift-must-gather-zzhm6/must-gather-vwt4f" Sep 06 06:34:31.254410 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.254125 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkkpt\" (UniqueName: \"kubernetes.io/projected/187d9d65-8478-4d4a-b79c-70f1555af1ee-kube-api-access-xkkpt\") pod \"must-gather-vwt4f\" (UID: \"187d9d65-8478-4d4a-b79c-70f1555af1ee\") " pod="openshift-must-gather-zzhm6/must-gather-vwt4f" Sep 06 06:34:31.254515 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.254481 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/187d9d65-8478-4d4a-b79c-70f1555af1ee-must-gather-output\") pod \"must-gather-vwt4f\" (UID: \"187d9d65-8478-4d4a-b79c-70f1555af1ee\") " pod="openshift-must-gather-zzhm6/must-gather-vwt4f" Sep 06 06:34:31.261763 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.261742 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkkpt\" (UniqueName: \"kubernetes.io/projected/187d9d65-8478-4d4a-b79c-70f1555af1ee-kube-api-access-xkkpt\") pod \"must-gather-vwt4f\" (UID: \"187d9d65-8478-4d4a-b79c-70f1555af1ee\") " pod="openshift-must-gather-zzhm6/must-gather-vwt4f" Sep 06 06:34:31.381738 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.381648 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzhm6/must-gather-vwt4f" Sep 06 06:34:31.650868 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:31.650844 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzhm6/must-gather-vwt4f"] Sep 06 06:34:31.652953 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:34:31.652918 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod187d9d65_8478_4d4a_b79c_70f1555af1ee.slice/crio-7d2ef7645e4decdcae5d90d8a0c5ca2f61ab0300cb37aa7aceb58446c16c434f WatchSource:0}: Error finding container 7d2ef7645e4decdcae5d90d8a0c5ca2f61ab0300cb37aa7aceb58446c16c434f: Status 404 returned error can't find the container with id 7d2ef7645e4decdcae5d90d8a0c5ca2f61ab0300cb37aa7aceb58446c16c434f Sep 06 06:34:32.000163 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:32.000063 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzhm6/must-gather-vwt4f" event={"ID":"187d9d65-8478-4d4a-b79c-70f1555af1ee","Type":"ContainerStarted","Data":"7d2ef7645e4decdcae5d90d8a0c5ca2f61ab0300cb37aa7aceb58446c16c434f"} Sep 06 06:34:33.010078 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:33.010040 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzhm6/must-gather-vwt4f" event={"ID":"187d9d65-8478-4d4a-b79c-70f1555af1ee","Type":"ContainerStarted","Data":"bce15152ef741135e8899a5bd0b39a06901ca92e0c9d02b6f0a1ff222d1a75a7"} Sep 06 06:34:33.010078 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:33.010095 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzhm6/must-gather-vwt4f" event={"ID":"187d9d65-8478-4d4a-b79c-70f1555af1ee","Type":"ContainerStarted","Data":"0b9781b21df37e1e06d88133ded7cb55d030d58bb1c482319bf4b68810c9c340"} Sep 06 06:34:33.025985 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:33.025925 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zzhm6/must-gather-vwt4f" podStartSLOduration=1.269631354 podStartE2EDuration="2.025870363s" totalImagesPullingTime="756.239009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 06:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:34:33.024061282 +0000 UTC m=+2438.422391751" watchObservedRunningTime="2026-09-06 06:34:33.025870363 +0000 UTC m=+2438.424200833" Sep 06 06:34:33.971777 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:33.971663 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c66ws_115882c1-6556-4feb-adc9-a5ddf5f36ba2/global-pull-secret-syncer/0.log" Sep 06 06:34:34.121703 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:34.121666 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9c7xb_faf01480-6d08-4434-822f-28b4815084d2/konnectivity-agent/0.log" Sep 06 06:34:34.193321 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:34.193287 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-32.ec2.internal_bd1de5c49e7422a298303b9a2c5c9a84/haproxy/0.log" Sep 06 06:34:39.511670 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:39.511631 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-zsfs2_76d99e41-d9a6-4a88-8960-00637b5b6efa/kuadrant-console-plugin/0.log" Sep 06 06:34:39.590103 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:39.590072 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-hzz62_89f58454-2203-4916-9a9f-76c089e3dcba/limitador/0.log" Sep 06 06:34:39.667069 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:39.667012 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-9kxg6_3c20a402-67b1-447a-91cd-2ef5c2d39255/manager/0.log" Sep 06 06:34:40.924844 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:40.924804 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fdb49626-f43e-4f1c-b4fc-28e68206df3a/alertmanager/0.log" Sep 06 06:34:40.950027 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:40.949997 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fdb49626-f43e-4f1c-b4fc-28e68206df3a/config-reloader/0.log" Sep 06 06:34:40.973784 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:40.973759 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fdb49626-f43e-4f1c-b4fc-28e68206df3a/kube-rbac-proxy-web/0.log" Sep 06 06:34:40.996167 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:40.996107 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fdb49626-f43e-4f1c-b4fc-28e68206df3a/kube-rbac-proxy/0.log" Sep 06 06:34:41.020752 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:41.020708 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fdb49626-f43e-4f1c-b4fc-28e68206df3a/kube-rbac-proxy-metric/0.log" Sep 06 06:34:41.048367 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:41.048184 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fdb49626-f43e-4f1c-b4fc-28e68206df3a/prom-label-proxy/0.log" Sep 06 06:34:41.073011 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:41.072019 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fdb49626-f43e-4f1c-b4fc-28e68206df3a/init-config-reloader/0.log" Sep 06 06:34:41.252029 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:41.251897 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-f7ngw_2630fd07-dd9f-426a-9d5a-b49afec19668/monitoring-plugin/0.log" Sep 06 06:34:41.441185 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:41.441152 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rcmg5_4a1ec7c7-c34f-4047-89fd-9cc56b7262a3/node-exporter/0.log" Sep 06 06:34:41.463456 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:41.463408 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rcmg5_4a1ec7c7-c34f-4047-89fd-9cc56b7262a3/kube-rbac-proxy/0.log" Sep 06 06:34:41.489647 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:41.489620 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rcmg5_4a1ec7c7-c34f-4047-89fd-9cc56b7262a3/init-textfile/0.log" Sep 06 06:34:41.790880 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:41.790846 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-nxrrr_2727d715-b3ee-442d-90eb-88773241e40f/prometheus-operator/0.log" Sep 06 06:34:41.813059 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:41.813031 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-nxrrr_2727d715-b3ee-442d-90eb-88773241e40f/kube-rbac-proxy/0.log" Sep 06 06:34:41.887213 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:41.887187 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-864c4594ff-tq5nd_d28c1df2-0e0f-4985-b32e-861722441838/telemeter-client/0.log" Sep 06 06:34:41.917142 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:41.917098 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-864c4594ff-tq5nd_d28c1df2-0e0f-4985-b32e-861722441838/reload/0.log" Sep 06 06:34:41.944034 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:41.944003 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-864c4594ff-tq5nd_d28c1df2-0e0f-4985-b32e-861722441838/kube-rbac-proxy/0.log" Sep 06 06:34:44.171540 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.171468 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/2.log" Sep 06 06:34:44.177842 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.177806 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8x82j_c23eec1b-1145-4e2d-9d9f-4b76157f40a0/console-operator/3.log" Sep 06 06:34:44.696185 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.696149 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw"] Sep 06 06:34:44.701838 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.701809 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:44.705967 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.705834 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw"] Sep 06 06:34:44.725209 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.725180 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-649cdb6d6d-2b5p7_1cbc8aa3-773f-4a89-9b2e-44f7b88dec42/console/0.log" Sep 06 06:34:44.761088 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.761057 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-hfnrt_87f80575-d356-4946-9532-39ba70525e7a/download-server/0.log" Sep 06 06:34:44.811741 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.811704 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d1b8f48-fec0-4e20-8f5e-87b880237704-lib-modules\") pod \"perf-node-gather-daemonset-p7bbw\" (UID: \"8d1b8f48-fec0-4e20-8f5e-87b880237704\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:44.812117 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.812094 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d1b8f48-fec0-4e20-8f5e-87b880237704-sys\") pod \"perf-node-gather-daemonset-p7bbw\" (UID: \"8d1b8f48-fec0-4e20-8f5e-87b880237704\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:44.812378 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.812360 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8d1b8f48-fec0-4e20-8f5e-87b880237704-proc\") pod \"perf-node-gather-daemonset-p7bbw\" (UID: \"8d1b8f48-fec0-4e20-8f5e-87b880237704\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:44.812545 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.812525 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvss9\" (UniqueName: \"kubernetes.io/projected/8d1b8f48-fec0-4e20-8f5e-87b880237704-kube-api-access-rvss9\") pod \"perf-node-gather-daemonset-p7bbw\" (UID: \"8d1b8f48-fec0-4e20-8f5e-87b880237704\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:44.812683 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.812667 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8d1b8f48-fec0-4e20-8f5e-87b880237704-podres\") pod \"perf-node-gather-daemonset-p7bbw\" (UID: \"8d1b8f48-fec0-4e20-8f5e-87b880237704\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:44.914224 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.914185 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d1b8f48-fec0-4e20-8f5e-87b880237704-sys\") pod \"perf-node-gather-daemonset-p7bbw\" (UID: \"8d1b8f48-fec0-4e20-8f5e-87b880237704\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:44.914419 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.914263 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8d1b8f48-fec0-4e20-8f5e-87b880237704-proc\") pod \"perf-node-gather-daemonset-p7bbw\" (UID: \"8d1b8f48-fec0-4e20-8f5e-87b880237704\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:44.914419 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.914289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvss9\" (UniqueName: \"kubernetes.io/projected/8d1b8f48-fec0-4e20-8f5e-87b880237704-kube-api-access-rvss9\") pod \"perf-node-gather-daemonset-p7bbw\" (UID: \"8d1b8f48-fec0-4e20-8f5e-87b880237704\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:44.914419 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.914312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8d1b8f48-fec0-4e20-8f5e-87b880237704-podres\") pod \"perf-node-gather-daemonset-p7bbw\" (UID: \"8d1b8f48-fec0-4e20-8f5e-87b880237704\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:44.914419 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.914332 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d1b8f48-fec0-4e20-8f5e-87b880237704-sys\") pod \"perf-node-gather-daemonset-p7bbw\" (UID: \"8d1b8f48-fec0-4e20-8f5e-87b880237704\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:44.914419 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.914351 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d1b8f48-fec0-4e20-8f5e-87b880237704-lib-modules\") pod \"perf-node-gather-daemonset-p7bbw\" (UID: \"8d1b8f48-fec0-4e20-8f5e-87b880237704\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:44.914419 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.914390 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8d1b8f48-fec0-4e20-8f5e-87b880237704-proc\") pod \"perf-node-gather-daemonset-p7bbw\" (UID: \"8d1b8f48-fec0-4e20-8f5e-87b880237704\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:44.914754 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.914467 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d1b8f48-fec0-4e20-8f5e-87b880237704-lib-modules\") pod \"perf-node-gather-daemonset-p7bbw\" (UID: \"8d1b8f48-fec0-4e20-8f5e-87b880237704\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:44.914754 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.914476 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8d1b8f48-fec0-4e20-8f5e-87b880237704-podres\") pod \"perf-node-gather-daemonset-p7bbw\" (UID: \"8d1b8f48-fec0-4e20-8f5e-87b880237704\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:44.921301 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:44.921271 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvss9\" (UniqueName: \"kubernetes.io/projected/8d1b8f48-fec0-4e20-8f5e-87b880237704-kube-api-access-rvss9\") pod \"perf-node-gather-daemonset-p7bbw\" (UID: \"8d1b8f48-fec0-4e20-8f5e-87b880237704\") " pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:45.020380 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:45.020343 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:45.314079 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:45.314030 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw"] Sep 06 06:34:45.316592 ip-10-0-135-32 kubenswrapper[2578]: W0906 06:34:45.316563 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8d1b8f48_fec0_4e20_8f5e_87b880237704.slice/crio-7fe739f9fb0af13b7a272788347b51b050e1d2714547894b091935c66b8c4e0b WatchSource:0}: Error finding container 7fe739f9fb0af13b7a272788347b51b050e1d2714547894b091935c66b8c4e0b: Status 404 returned error can't find the container with id 7fe739f9fb0af13b7a272788347b51b050e1d2714547894b091935c66b8c4e0b Sep 06 06:34:46.116299 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:46.116260 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" event={"ID":"8d1b8f48-fec0-4e20-8f5e-87b880237704","Type":"ContainerStarted","Data":"0cf7d26e578194d0ea49befe9cd8db6710e50e30ea7cabbde858e217ad7345b8"} Sep 06 06:34:46.116480 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:46.116375 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" event={"ID":"8d1b8f48-fec0-4e20-8f5e-87b880237704","Type":"ContainerStarted","Data":"7fe739f9fb0af13b7a272788347b51b050e1d2714547894b091935c66b8c4e0b"} Sep 06 06:34:46.116480 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:46.116436 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:46.131868 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:46.131806 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" podStartSLOduration=2.131788648 podStartE2EDuration="2.131788648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 06:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 06:34:46.129963459 +0000 UTC m=+2451.528293946" watchObservedRunningTime="2026-09-06 06:34:46.131788648 +0000 UTC m=+2451.530119116" Sep 06 06:34:46.590261 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:46.590228 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zmlqw_fee5736d-8984-42af-9121-8fe5d123a3ce/dns/0.log" Sep 06 06:34:46.611621 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:46.611586 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zmlqw_fee5736d-8984-42af-9121-8fe5d123a3ce/kube-rbac-proxy/0.log" Sep 06 06:34:46.636889 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:46.636819 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qcvmz_e97d5427-3253-45fd-8a75-b0f4a8dd0452/dns-node-resolver/0.log" Sep 06 06:34:47.247126 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:47.247092 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6fw4z_bddf45b0-9199-4a9d-aee3-ca253d8a9c56/node-ca/0.log" Sep 06 06:34:48.188414 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:48.188385 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv_9881c8ee-3e68-4e99-a17c-198451cbf660/istio-proxy/0.log" Sep 06 06:34:48.542491 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:48.542449 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rkjhl_b1e56087-4754-48ce-9c1c-8f16b57c74b1/istio-proxy/0.log" Sep 06 06:34:48.555915 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:48.555874 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rkjhl_b1e56087-4754-48ce-9c1c-8f16b57c74b1/istio-proxy/1.log" Sep 06 06:34:48.649975 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:48.649941 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-99f956f9b-9d6ht_c9644473-eedd-4380-a23d-ecf5b0c85f10/router/0.log" Sep 06 06:34:49.529683 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:49.529657 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-trktn_df6097ca-4ae8-47d6-918b-189f75f655cd/serve-healthcheck-canary/0.log" Sep 06 06:34:50.005119 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:50.005039 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-6jcbw_50fa393c-a0bf-4082-8040-8018f2ba260c/insights-operator/0.log" Sep 06 06:34:50.006512 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:50.006488 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-6jcbw_50fa393c-a0bf-4082-8040-8018f2ba260c/insights-operator/1.log" Sep 06 06:34:50.104428 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:50.104401 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-llpf2_c05aa74a-9686-4a1c-a26e-68ef0f755859/kube-rbac-proxy/0.log" Sep 06 06:34:50.129342 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:50.129312 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-llpf2_c05aa74a-9686-4a1c-a26e-68ef0f755859/exporter/0.log" Sep 06 06:34:50.151659 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:50.151634 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-llpf2_c05aa74a-9686-4a1c-a26e-68ef0f755859/extractor/0.log" Sep 06 06:34:52.138284 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:52.138247 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zzhm6/perf-node-gather-daemonset-p7bbw" Sep 06 06:34:52.542448 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:52.542402 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-ssg8b_63866f27-a454-4498-b1ab-14973b58a81e/manager/0.log" Sep 06 06:34:52.717051 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:52.717018 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-xslf6_c525d127-0f22-4fc8-84e8-58a618a0dbaa/manager/0.log" Sep 06 06:34:52.889031 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:52.888926 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b897cd959-rsx7c_3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266/manager/0.log" Sep 06 06:34:52.910156 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:52.910124 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b897cd959-rsx7c_3e0f2c6b-5ca7-45e5-8d6d-cce3eba84266/copy-manifests/0.log" Sep 06 06:34:54.412510 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:54.412469 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6fb67bfd69-gss5n_0c02d25d-d4c5-49c7-8a4a-d9c80f7105fa/manager/0.log" Sep 06 06:34:54.482502 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:54.482469 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-z6k5b_35cf5209-0bf5-426b-840d-8870ea811702/openshift-lws-operator/0.log" Sep 06 06:34:55.215801 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:55.215743 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv_9881c8ee-3e68-4e99-a17c-198451cbf660/istio-proxy/0.log" Sep 06 06:34:55.534115 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:55.534082 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rkjhl_b1e56087-4754-48ce-9c1c-8f16b57c74b1/istio-proxy/0.log" Sep 06 06:34:55.546960 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:55.546929 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rkjhl_b1e56087-4754-48ce-9c1c-8f16b57c74b1/istio-proxy/1.log" Sep 06 06:34:55.614630 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:55.614598 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-99f956f9b-9d6ht_c9644473-eedd-4380-a23d-ecf5b0c85f10/router/0.log" Sep 06 06:34:55.797294 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:55.797202 2578 scope.go:117] "RemoveContainer" containerID="0eaf3d97bbb66a2160420f8d2fe01701109564ffd573d106f70374a9d4accffb" Sep 06 06:34:57.177042 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:57.177012 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-fwnnv_9881c8ee-3e68-4e99-a17c-198451cbf660/istio-proxy/0.log" Sep 06 06:34:57.524237 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:57.524158 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rkjhl_b1e56087-4754-48ce-9c1c-8f16b57c74b1/istio-proxy/0.log" Sep 06 06:34:57.539893 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:57.539863 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rkjhl_b1e56087-4754-48ce-9c1c-8f16b57c74b1/istio-proxy/1.log" Sep 06 06:34:57.615066 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:34:57.615026 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-99f956f9b-9d6ht_c9644473-eedd-4380-a23d-ecf5b0c85f10/router/0.log" Sep 06 06:35:00.119375 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:35:00.119340 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-ff6vt_74d86613-a3ce-4adf-bf86-934052abad09/kube-storage-version-migrator-operator/1.log" Sep 06 06:35:00.120097 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:35:00.120075 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-ff6vt_74d86613-a3ce-4adf-bf86-934052abad09/kube-storage-version-migrator-operator/0.log" Sep 06 06:35:01.071224 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:35:01.071194 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4b45g_557fb632-fd3a-4b82-b25a-ba2e25072337/kube-multus-additional-cni-plugins/0.log" Sep 06 06:35:01.093171 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:35:01.093140 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4b45g_557fb632-fd3a-4b82-b25a-ba2e25072337/egress-router-binary-copy/0.log" Sep 06 06:35:01.115973 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:35:01.115945 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4b45g_557fb632-fd3a-4b82-b25a-ba2e25072337/cni-plugins/0.log" Sep 06 06:35:01.137601 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:35:01.137574 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4b45g_557fb632-fd3a-4b82-b25a-ba2e25072337/bond-cni-plugin/0.log" Sep 06 06:35:01.159731 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:35:01.159707 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4b45g_557fb632-fd3a-4b82-b25a-ba2e25072337/routeoverride-cni/0.log" Sep 06 06:35:01.181378 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:35:01.181347 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4b45g_557fb632-fd3a-4b82-b25a-ba2e25072337/whereabouts-cni-bincopy/0.log" Sep 06 06:35:01.203255 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:35:01.203225 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4b45g_557fb632-fd3a-4b82-b25a-ba2e25072337/whereabouts-cni/0.log" Sep 06 06:35:01.592333 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:35:01.592296 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dm4ww_9a837857-cbb6-411d-8d9c-1af494b92d13/kube-multus/0.log" Sep 06 06:35:01.745436 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:35:01.745404 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-w74pq_f36acd55-4ce4-4bdb-9518-cca261651a19/network-metrics-daemon/0.log" Sep 06 06:35:01.765951 ip-10-0-135-32 kubenswrapper[2578]: I0906 06:35:01.765889 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-w74pq_f36acd55-4ce4-4bdb-9518-cca261651a19/kube-rbac-proxy/0.log"